Ep. #11, Backstage with Lee Mills and Matt Clarke of Spotify

Marc Campbell: Hi, I'm excited about today's episode of the Kubelist Podcast.

My guests today are Lee Mills and Matt Clarke from Spotify.

And we're going to talk about Backstage, one of the newer CNCF Sandbox projects that was donated by Spotify.

Welcome Lee and Matt.

Lee Mills: Hey, thanks for having us.

Matt Clarke: Nice to be here.

Marc: Great, before we get started I'd love to give you each an opportunity to introduce yourselves. Lee do you want to start?

Lee: Sure, so my name is Lee. I'm the Engineering Manager for Backstage.

So I look after both Backstage, the open source project and our internal kind of instance that we run in a Backstage at Spotify.

Marc: Great and Matt.

Matt: My name is Matt Clarke and I'm a Senior Infrastructure Engineer at Spotify.

I work on the deployments team and mainly that's focused towards deploying backend services and websites to Kubernetes.

Marc: Great, so what's Backstage? What does the project do?

Lee: So Backstage is our kind of turbocharged developer portal, if you will.

So it's a place you can go to find all of your things or find all of the things at your organization.

It's a place where you can go where you manage and maintain those services and find out all kinds of bits of information around them.

So really it's our one stop shop for everything to do with services, tooling, and documentation.

Marc: OK, can you give me an example of like the type of a service that I might put in the Backstage?

Lee: Sure, I mean it's really anything that you're wanting to build out.

So as an example, we have Backstage itself is in Backstage which is kind of meta, but it allows us to go and inspect like the backend of Backstage and see how it's running, check on its health, do operations on it.

Same with the front end of Backstage and then bubble up all kinds of information that's related to Backstage around that.

So we can go in and we can see, for example, our costing sites.

We can see how much it's costing us running these services.

We can find out information about the health of the clusters that it's running on.

And that's something that I know that Matt's been heavily focused on.

I don't know if you want to jump into it and just talk a little bit about the things you've been working on in Backstage on those services.

Matt: Yeah, sure.

So we recently opened sourced a Kubernetes plugin for Backstage and the idea behind this Kubernetes plugin is that it's a Kubernetes monitoring tool, which is designed for microservice owners and they can use it to track the status of their services deployed onto Kubernetes.

And that's regardless of what cloud their Kubernetes cluster is running in or how many clusters they have.

So that they can go in and see if Backstage has detected any errors in the Kubernetes objects.

You can also see information like how close you are to your auto-scaling limits.

Marc: That totally makes sense.

I mean, I'm looking at the Backstage site right now and you describe it as an open platform for building developer portals.

I'd love to dive into that description of developer portal, because Matt what you just described, you know, you talk about service owner and actually getting into like monitoring the application when it's running in production.

How do you think about like how Backstage and the capabilities that it's running as that Kubernetes add-on kind of move into the whole monitoring and telemetry ecosystem that exists in Kubernetes?

Is it complimentary? Is it potentially a replacement? How does it fit?

Matt: So I see it as complimentary. I think a lot of the tools in Kubernetes are very holistic.

So if you look at the Kubernetes dashboard, it drops you in, it gives you kind of complete access to look at everything on a Kubernetes cluster.

But whenever we were thinking about what our developers at Spotify needed, we realized that actually they didn't really care about looking at things at a cluster level, they cared about looking at things at their service level.

So instead of giving the developer lots of information about everything that's running on a cluster we go through our clusters and look only for their service and then aggregate that information in Backstage just so that they can have this focused overview of their service from one place.

Marc: And so I assume like, you know, this is built on the organizational patterns and the way that you're trying to empower developers at Spotify to own the services that they're running.

Lee: That's right and that was actually one of the really interesting challenges that we had when we were kind of building out Backstage was basically because of the distributed teams model and that distributed ownership that we have at Spotify, we had to kind of provide a Backstage in a way that it can be extended really, really easily.

And those extensions or plugins as we call them need to be owned by the teams.

So in Matt's case and the Kubernetes plugin that's owned by Matt and his team.

Those guys know better than anybody else about Kubernetes and about the infrastructure at Spotify.

It doesn't make sense for like the central Backstage team if you will, it doesn't make sense for them to own that.

So we ended up coming to this distributed ownership model that enables teams to build out the use cases that they need to support without having to go through that central team and that central bottleneck.

So it kind of grows organically at this stage at least internally for Spotify.

So, which is really cool to see.

Marc: Yeah, that totally makes sense.

And I love that model, you know, we're a small startup, we're around 40 people right now, but we follow that pattern and I think it's really common today where you have maybe an SRE team or a platform team that's responsible for owning the platform itself.

But, you know, just making sure that the developers who want to like ship something to that infrastructure have the support they need.

Like minimizing repeated efforts and redundant efforts to make sure like there's efficiency in the platform.

Matt: I think that's one thing we really wanted to do when we created this Kubernetes plugin.

So once we started having multiple Kubernetes clusters and we had developers deploying their services to multiple clusters whenever they wanted to like see the current status of their application running on those clusters, they would have to use Kubectl, have multiple contexts as they're constantly switching contexts and they can't really find a tool that gives them an aggregated view of their service no matter where it's running.

Lee: You've touched on a really important point there as well Matt because that kind of context switching that cognitive load, how it's happening in Kubernetes happens across the board with all of the different tools, all of the different use cases.

And that's where kind of Backstage itself trying to be that central place, trying to be consistent in the way it's used and the way it looks reduces that kind of cognitive load for people be that in Kubernetes or be that in any of the other areas that we're supporting through Backstage as Spotify.

So it's the same, I guess, the same problem there that you were having with Kubernetes and using Kube Cuddle zoomed out to kind of like Kubernetes plus all of the other kind of areas that we use it in BHCICD or testing, or like documentation discovery even.

Marc: That makes sense.

There's like this tricky line to navigate around, you know, you want consistency and uniformity but you want developers to have the freedom to choose and build what they want.

Because look that's why you're hiring these creative engineers, you know, to come in with new ideas and figure it out.

But you also have to worry about compliance and regulatory reasons or just not every problem needs to be invented every time a new service is deployed, right?

Lee: Exactly, exactly.

Like the two things that stand out for me on that side of things is one like providing, through Backstage providing the tools to make discoverability really easy.

So the first question is, you know, as an engineer I guess I'd say, right, well, we need to solve this problem.

Has anybody else solved it?

And previously, you know, that could have been jumping into random Google docs or it could've been jumping into Confluence or Wikis or like all over the place trying to find out if someone has already solved this problem.

So the first part was let's solve the discoverability bit.

And then the second part is if people need to then go and build something, give them really easy intuitive tools to be able to do that.

And with those tools we have a software templates where basically you can build in those best practices.

So if there are compliance things or if there are security concerns or like all of those kinds of fundamental bits, you can build them into templates.

And then when engineers or when teams come to use those templates they get those best practices built in from the get-go.

So again, the kind of we're zooming out.

So they, rather than spending the time and focusing on the plumbing and the things underneath they get to focus most of their time on helping their own customers or solving the use cases that they're trying to solve.

Marc: Yeah, I can imagine that that pays off on multiple levels right?

Just, you know, both in terms of developer productivity, compliance, you know, you can create, you're not enforcing uniformity, but because that's the default, a lot of folks will use that but probably even just new developer onboarding when you bring somebody new on like, there's both the consistency and the ease of like spinning up a new service probably pays off every day.

Lee: Yeah, totally.

And specifically with the onboarding there that's something that we do, every engineer that joins kind of jumps into this kind of camp if you are with other people joining the company.

And in the first few weeks you'll be building something and you use Backstage to do that.

So you go in and you get the kind of the documentation the support, the help free Backstage, but then you exploring like the templates and using that to spool up a service and then you get the freedom of right, that gives you like a basic scaffolded service or component and then you can basically go nuts.

You can go in and customize and build anything that you want from that point kind of thing, knowing that you've got the support and the tooling and all of these kind of functionalities around you.

Marc: Yeah, let's just shift gears for a minute and just talk about the architecture of Backstage itself, because you've talked about and provided some examples around templates, software templates and monitoring the service in production.

But is this just what Backstage does it's all built in or like, I think there's like plugins and a pluggable model, I'd love to like dive into a little bit more.

Lee: Yeah, absolutely. So Backstage at its core is three core things.

It's the service catalog which is kind of like your inventory. It's where you can go and find services.

You can find out about ownership, you know, the teams that own services and who to contact and all of that sort of stuff.

Then we have the software templates.

So again, you can build new templates in to suit your organization, to suit your teams but you can build in best practices as you go.

And the third then is the discoverability piece and the extensibility, which comes with the plugins.

So Backstage can really morph and change to fit your needs.

Like Backstage itself is relatively compact compared to like the wealth of things that you could build through plugins and that side of things. And a concrete example of that, like 85% of Backstage internally at Spotify is built by other teams, it's not built by the core Backstage team.

And that just goes to show kind of the power of these plugins and hopefully the simplicity of it with so many people using it.

But I think Matt, you're probably better to talk to that side of it because you're on the other side of that, you've had to use Backstage to build plugins and that side of it.

Matt: Yeah, so I think initially whenever we wanted to give developers visibility about their deployments and the current status of their application through Backstage, this was in the pre Kubernetes days at Spotify.

So most developers at that point would have used Helios to deploy their service which is our open source container orchestration system.

So we actually built a plugin that allows you to deploy different versions of your application to Helios in different regions that we were deployed to.

And as our Kubernetes experience kind of developed and we started adopting Kubernetes more and more we transitioned that plugin from being the kind of Helios plugin to being our internal Kubernetes plugin and as we developed it I think developers at Spotify really got a lot of use out of being able to visualize their Kubernetes deployments in one place, getting that kind of instant feedback.

So whenever Backstage was open source we really wanted to contribute with a lot of the learnings that we had from that.

But one thing we really wanted to do was not to open source the internal Spotify Kubernetes plugin.

We wanted to build a plugin with community involvement that would work for everyone.

So that's kind of what we've done in the open source, Kubernetes plugin for Backstage.

Marc: That's great, that makes sense.

Kind of off topic a little bit, Matt but a question for you like, I remember, you know, when Helios was announced and you shared it and that's great and really cool.

I'm curious, like is Spotify Arlene on Kubernetes at this point?

Matt: I think that we get to really appreciate how fantastic Kubernetes is.

And lots of developers at Spotify are using Kubernetes a lot.

Helios still plays a part in our infrastructure, but I can't really speak as to the future of it.

Marc: Good. If I want to write a plugin, what skills do I need? What languages do I write?

Like how much do I have to understand the Backstage architecture versus just like solving the problem?

Like what's the effort to re write a new plugin, I guess?

Lee: So, I mean, Backstage itself it's React TypeScript with a NodeJS backend.

The frontend of the application, we basically took what we had in internally and kind of rolled out with the open-source project.

The backend was something that we kind of wanted to take another look at because the backend of the project internally is kind of woven into a lot of Spotify specific stuff that isn't really of much used in external.

And by that, I mean, you know, for example not everybody is going to want a plugin that is focused on, you know, creating or managing Spotify playlist.

So what we did for the backend is we took a look at the CNCF landscape and a bunch of other open source projects and trying to kind of identify where engineer skillsets in our community tended to lie and what were the frameworks?

What were the languages? What were the things that they started that they were commonly using?

And that's how we settled on the NodeJS backend?

Well, that means for when it comes to run a plugin to get back to your question, sorry is we have a command line that comes with Backstage.

So it's a simple command to kind of scaffold your first plugin if you will.

And what that basically does is kind of building all the hooks and bits and pieces for Backstage and then gives you a blank slate.

We then provide like a storybook for front end kind of components and things like that that you can reuse, it's based off material UI.

So again, another popular, quite common framework there.

And then hopefully we provide a bunch of different documentation and then the open source community there to support you.

So in theory to get going with your very first plugin, to get something that is kind of ready for you to start building out really simple, get it done in minutes.

And then basically you're jumping into a TypeScript plugin at that point.

And for the backend of your plugin you can connect that up to whatever services or things that you have already existing.

For example, with Kubernetes, I would imagine and Matt will be able to speak more on this but I guess for that plugin there's a backend somewhere that was either pre-existing or is a different service to Backstage that can hook up to.

Same with like the costing size plugin that we have or like the Jenkins plugin or any of the others.

Marc: That's great. I mean, look, I love that.

That makes total sense, like lower that initial learning curve, make it really accessible.

So it's easy to make Backstage do what you actually want.

Lee: Yep, exactly.

Marc: So Backstage is open source and Spotify donated it to the CNCF recently.

I think it was just towards the middle or end of 2020.

Were either of you involved in that decision to donate it to the CNCF as opposed to just leaving it as an open source project?

I'd love to understand like the thought process that you guys had to get there.

Lee: Sure, so I was involved in a few of the conversations.

I'm definitely not the decision maker if you will but I was definitely involved in a few of those conversations and a lot of it revolved around a couple of things.

So Backstage is solving a problem at Spotify.

The discoverability problems that we were having, the fragmentation of multiple teams spooling up the tools to do the same things and the ownership side of things.

And we went out and talked to a bunch of different end users in the community and found that there were similar kind of problems or use cases that other organizations were having.

So that kind of played into the general opensource decision for Backstage.

With the CNCS we saw as an opportunity to get involved more and more with the community for one, and being involved in that community has been amazingly useful because the Backstage core team, I mean, right now there's four people working on Backstage for opensource.

But we have a community of, I don't know how many we just had a community event and over 100 people turned up to it.

So all of those different ideas, all of those different perspectives and points of view have really contributed to drive Backstage to be better.

And there's been a bunch of things creating anything open source that we're either adopted or adopting and Spotify.

So one, the power of the community trying to make Backstage better.

And two, we really believe Backstage solves a problem and it's a problem that many people have.

So being part of the the CNCF gives us all kinds of connections and visibility and exposure for a wider audience.

Again one we'd love to welcome them as part of community, but two we hope that Backstage can solve something for them too.

Marc: Lee, have you seen that since donating it to the CNCF?

Have you been able to recognize some of that value and say, wow like this is actually increasing community engagement or any of those other metrics?

Lee: Sure, so, I mean, obviously community engagement is actually it's surprised as all.

When we dropped Backstage into open source almost a year ago now, we're coming up on the one year anniversary of that.

And we were kind of doing it in a way that we said we know it's not perfect right now for the opensource but we want to build it together.

So we decided to kind of release it to opensource a lot earlier actually than we were originally planning.

And we started out with an initial community that is just then just grown from strength to strength to strength.

And we're seeing kind of contributions go up week on week.

I think it's every week, we're roughly adding two contributors to the project right now, every single week.

And it's amazing to see. And then the flip side we've had a lot of interest from all kinds of different types of organizations actually reaching out, interested in the project.

Hey, we've got these problems, hey, we've got these use cases.

Some of which we've seen we've experienced at Spotify.

Some are completely new, which is really exciting to see if like we had ideas for this project and now it's been used in all of the different use cases that we hadn't even considered.

And on that side of things as well that side is growing too.

I think we're up to about 19 organizations that are publicly in the repository.

And then we're at over 200 I think it is that we've been kind of chatting with or discussing use cases, helping people to try and solve the problems that they're having too.

Marc: That's great. I love that story too.

Like, you know, as a developer, you know, thinking about open sourcing something, it's easy to say, oh, but you know, it has these problems, so it's got these challenges, let me clean this up and then open-source it.

And, you know, being willing to open source it with the problems is just like a, you know, if you get adopters and community users of it that early, you know, even with those known problems and they're helping you fix them, it's clear that there's project market fit and people are like you're solving a real problem out there.

Lee: Yeah absolutely, absolutely.

Marc: You know, you mentioned that our Backstage was created to solve a problem that Spotify was having.

I'd love to like go into that a little bit more, I have a couple of questions around it, you know, like what size was Spotify when you had these problems and like, was there some trigger, some external factor that was happening in the ecosystem that made these or was it just organizational size that was causing it?

Lee: Sure, so I don't actually know the like the number specifically as to what was the size of Spotify at the time.

It was four and a half ish years ago when the very first kind of parts of Backstage came to life.

It wasn't called Backstage back then obviously it didn't do everything that Backstage does now.

I think initially it was mostly focused on kind of inventorying services and identifying ownership originally.

And that kind of grew out of, as we were talking about earlier the ownership model that Spotify has.

We see real real value in teams owning their own destiny, owning the services end to end that they have.

And as Spotify was growing and these teams were, you know, we were getting more and more kind of teams come on board, we started to hit these problems around fragmentation, around discoverability, around kind of context switching in the cognitive overload that were kind of impacting engineers in their day to day work.

And then as we kind of grew further obviously systems started to get more and more complex.

And all of a sudden, everybody had to be an expert in everything to be able to do anything before you even start kind of like what you should be doing that day.

So they were kind of like the fundamental issues.

I know originally, as I say, we kind of started out in this inventory pricing side of things.

And then as we kind of started to realize that, hey, there's all of these other problems.

So there's the fragmentation, this is the discoverability. We started to build things around it.

And namely that came in through kind of these plugins.

This idea of being able to surface all of these different services or all of these different tools like Kubernetes that we've been using through a single kind of almost a single pane of glass, if you will.

And in doing that engineers and teams realized like if they use one tool or if they're looking at one service they know that they can go and look at something else and it's going to be in a kind of a similar consistent way that they kind of discover and view and interact with services.

And that reduces that kind of context switching a little bit, it reduces the cognitive load a little bit.

And then when you start to get a kind of, almost not a critical mass, but there is a kind of turning point.

So sorry if I rewind a little bit.

Earlier in the Backstage process, we were trying to obviously build out some plugins and kind of get things going.

And what we kind of realized and the same way that now, when I talked to other organizations thinking about adopting Backstage, it's really about finding one or two pieces of value.

So one or two kind of real problems that you can solve in Backstage.

And typically they'll come in the form of, you know, maybe the, let's say the catalog and a few tools or a few plugins around those services in the catalog where it gives you a use case.

Something that an engineer or someone in an engineering team will go and do every day and it solves that problem for them.

And when teams and engineers and the rest of the organization start to see that value you start to get a kind of a poll and people start to kind of say, right, well, we need to build this tool, I'm not going to go and build it over here, I'm going to go to Backstage because that's where everything else is.

And if I add this in that extends that value. It solves this new use case that I have, or it extends that use case that people have already been working on.

So you kind of get this tipping point where then all of a sudden people are starting to add to Backstage.

And when that starts to happen you start to see kind of productivity increase.

You start to see, you know, these discoverability issues start to go down.

You know in a simple thing if I put the documentation for my service with my service and then bubble that up through Backstage, all of a sudden I'm not having to deal with two different tools.

Or if I want to know more about the service I just go and look at the service and then there's the documentation and there's the plugins and tools to kind of maintain it around it and there's the monitoring and that sort of stuff around it too.

Marc: I have a question for you, Matt.

Like if I'm, you know, a really small startup I might just be, you know, you haven't raised any funding yet or really early and just a few people and just running Kubernetes on GKE or EKS and managed Kubernetes that--

I don't have an SRE team, I don't have a platform team, Is that too early for the value that you're getting out of it or would it be great just to like start that way?

Matt: I think it would be good to start because one of the benefits of this Kubernetes plugin is that it works regardless of where your Kubernetes cluster is hosted.

And therefore, if you're a small startup, you know, you might want to go to the Kubernetes cluster that is hosted by the cheapest provider.

So this lets you have that flexibility from an early stage, but also, you know, it doesn't limit your scalability in any way. So you can use it.

If you're a small team, you can use it, if you're a large organization I don't think there is like a lower limit.

Marc: OK, that's great.

Talking about the ecosystem a little bit, you know, you talked about like you're seeing new contributors and new use cases and new organizations, you know, constantly using it and that's great.

It's also kind of spawned this like world where there's actually some commercials startups built around some of the, like, the developer platform but even specifically around like the Backstage implementation.

Do either of you have any thoughts on that startup ecosystem that's coming out around the Backstage world?

Lee: From my perspective, one, it's really exciting.

I mean, it's awesome to see that people are kind of taking a look at Backstage and then kind of seeing what it solves and then kind of making that decision to, hey, we're going to try and build something around this where, you know, we believe in what this project is trying to do so let's build something around it.

And for me, I mean, one is fantastic that we've been able to provide something that can do that for other people who want to, you know, kind of build a startup around it.

And I think it's super healthy and it's something you see in other kind of CNCF projects or other open source projects where you get these kinds of startups starting open providing either hosted solutions or like different takes or support, or like all of these different things around the project.

And it's something we've seen like with Backstage we've seen multiple startups now doing things around Backstage and providing that support from the community.

And the really great thing is those startups are also super active in the open source community too.

So they're contributing things back, they're getting involved in the conversations there, they're helping the community to kind of guide the project and then they're really kind of getting involved and wanting to learn more about Spotify, about the project itself, about where it's going and all of these kinds of aspects.

So it's, if nothing else, I think it's really exciting and it's awesome to see and I didn't expect it to be honest.

I mean, this is me personally, this isn't like, this is not Spotify talking if you will right now but me personally, I really didn't expect to see that within the first year of the project going live.

So it's really exciting.

Marc: Yeah, that's really cool.

I mean, look as developers, like, you know, we'd love when people get value and see something out of the project and it's clear the Backstage is doing that right now but like just to like spawn and tire companies based on that it's like a different level than just adding value to their Kubernetes infrastructure.

Lee: Yeah absolutely.

Marc: So we've talked a lot about what Backstage does today. I'd love to kind of look forward a little bit what's on the roadmap?

What should we look forward to in the next, you know, the next couple of releases or in 2021 here for Backstage?

Lee: Sure, absolutely. I mean, right now we're heavily focused on stabilizing.

We're focused in on the service catalog, we're focused on the software templates aspects and kind of the core underpinnings, if you will.

We're almost at a year in our open source journey and we really want to make sure that we're doing a really good job and we're giving confidence and everything to adopters so that they can jump in, they can take Backstage on board and they know that it's going to, you know, we're invested, we're maintaining, we're dedicated to this.

So we're really focusing on that stabilization right now.

We've got some really cool stuff coming up around the software templates as we speak right now making them a little bit more extensible and giving organizations that adult Backstage more opportunities to kind of make it fit for them.

We're also thinking about search and that discoverability side of stuff.

We've got some great stuff there already around our tech doc solution, our docs like code solution.

We're really wanting to dive into taking that further and also kind of investing in the search side of things and really bringing that to the platform and helping to improve that discoverability.

And then kind of going forward next, we really want to get into kind of the building the ecosystem around it if you will.

So right now you can go and have a look.

We have a bunch of plugins up on the Backstage website where, you know, the community has contributed or Spotify open-sourced a few as well and we really want to get into to that side of it.

Now it's really about like what are the things that we need for people to adopt?

How do we make it, you know, that super simple?

How do we make the kind of plugging creation and maintenance and that side of things really awesome for people to use?

And then really let's see whether the community kind of takes us a little bit.

I'm kind of really really interested to see what plugins come along, what kind of different customizations to Backstage people do.

And then specifically, I mean, Matt I'll let you talk more about it, but one thing of that kind of platform agnostic in Kubernetes.

I'm really, really interested to see all the different kinds of things that people do there and kind of how can we extend the Kubernetes support and all that side of it.

But anyway, Matt, what about you?

Matt: Yeah, I'm really interested to see how the community use the Kubernetes plugin and I'm also interested to see what custom resource definitions they want support for. I think that's definitely something that we want to do.

So for example, you could use Argo roll lights open-source CNCF project to automated Canary analysis or blue-green deployments and that comes with a custom resource definition of CRD.

And traditionally those sorts of objects, haven't really been very visible through Kubernetes monitoring tools.

And I think through Backstage, we might be able to provide a really good UI for those custom resource definitions and provide extensibility within the Kubernetes plugin itself.

Marc: Is the Kubernetes plugin in Backstage, is it itself like an operator in a CRD?

Matt: No, it's not an operator, it doesn't have its own CRDs.

It access is just from the Kubernetes API perspective.

It doesn't have to run inside your clusters which is a positive for installing it.

It makes it installation quite simple.

Marc: Yeah, I'm sure that helps kind of lower that security, initial security like review that needs to be done 'cause you know, operators are cluster admin level, so.

Matt: Yeah, definitely.

So at the minute, everything is just read only and you need access to various Kubernetes objects such as deployments, services, conflict mops but yeah, it definitely will because like I said, you only need to read only permissions

Marc: And go diving into the weeds for just a minute.

Like if I want to try that out but like limit it to namespaces I'd be able to use Kubernetes are back to give it a role that controls what it can access but the plugin would still work I assume?

Matt: So namespace permissions is currently not supported.

So you do need cluster read on the access but I think that's actually something that's been brought up in community discussions that'd be really interesting.

Marc: Cool, so going back to the roadmap a little bit.

You know right now Backstage is a Sandbox project and for anybody who's not super familiar, you know, the CNCF goes from Sandbox to incubation to a graduated projects.

Have you started to give any thought I know it's still early, it's only been open source for a year but have you given any thought to what the milestones are or what the requirements are going to be and maybe even timeline for going after the incubation level?

Lee: I mean, of course we've thought about it. I think that was, you know, that was one of the conversation as we Sandbox, it was like, right, what's next?

How do we get to the patient? So it's definitely come up in conversations.

I don't think we've nailed down exactly the milestones just yet.

For me, the things that we're focusing on right now around you know, are we seeing adoption of it?

If we are, what are the things that people are needing more off or where are the things that we're not doing?

And I think as we start to see more and more people solve their own kind of use cases with Backstage that will build a confidence more and more in the project that we're in a state where we're ready to kind of go to incubation.

So it's perhaps early at the minute.

We haven't nailed on those specific milestones just yet but it's certainly like it's rooted in that community.

So that's why we're focused right now.

Marc: Great and so as other CNCF companies or developers or users like what's the best way to get engaged with the project other than just running it?

Like, is there a specific type of feedback you're looking for?

Is there help on certain types of, you know, like issues or like how would we get involved?

Lee: Sure, so we do have a pretty vibrant community on GitHub right now, which is awesome.

Everything that the core team is doing, we're doing in the open through GitHub and we also have a Discord server as well which you can get to from the main Backstage site.

Up on GitHub everything where we either us or the community of how to ideas where we're not focusing on it right now is basically marked up as kind of how it wanted.

So you can do a quick search for that label.

If you're interested in particular plugins there's a label for that to on GitHub.

So you can go in and see there's a bunch of plugins there that people have suggested or they have used cases that they want to solve with so that's a another one to account for.

And then really it's getting into the community on Discord or join us at one of our Backstage community meetings that we have once a month.

All of the details for the community meetings are up on GitHub and I say Discord is linked to from the main Backstage website.

Marc: That's great and Discord is interesting.

Like, I've seen a couple others CNCF projects recently starting to talk about moving away from Slack channels into the Discord.

Like you're getting good traction, get adoption from community members inside Discord.

Lee: Absolutely, absolutely.

I mean, Discord has been great just because of, you know, the flexibility of being able to have all of these different community channels.

We can also with the audio or video channels there and stuff like that, we can host like bits of meetings if we want to or jaw drop easily into a quick call with people to discuss through things.

But I don't think Discord has slowed us down, it's been a great enabler for us.

Marc: That's great and so you have regular community meetings and everything that you're working on is being done out in the open Spotify.

Like you don't have any internal conversations.

You said Lee, that you had four full-time developers at Spotify working on the project, but like their work is all out in the open right now?

Lee: All the work is out in the open.

I mean, the engineers on the project hangout in Discord all day.

As an example, we have a voice channel where they're communicating all of the day.

So some of that conversation is done just between the maintainers there or the engineers there but usually that's a case of they'll discuss things back and forth, and then they bring it out into the community either through an issue in GitHub or into like the Discord chat there and just chat with people.

So yes, there are some conversations that happen behind the scenes, but usually that's just discussing ideas from the community or things that we want to bring to the community.

Marc: That's fun, it sounds like a fun job too. Just being able to work totally out in the open.

Did this team come out of existing Spotify engineers or did you hire and specifically recruit for open source engineers?

Lee: Yeah, so actually it came out of the original team that looked after Backstage internally at Spotify.

And what we ended up doing is we grew that team with the intention of kind of splitting it if you will.

So we have from the original team, we have half of that team now focused, purely on open-source.

They care about the community, that's their customers, that's their ownership domain.

And then we have the other half of the original team.

Now look after the Backstage instance internally at Spotify and they're kind of operating as a customer of the open source obviously, and their kind of customer their domain is internal Spotify.

Marc: Great there, that totally makes sense.

So thanks Lee, thanks Matt from Spotify, you know, for talking about Backstage, sharing the roadmap and I think anyone listening should probably go check out the docs and probably install it and get started if they have any of the problems that you mentioned today.

Lee: Yeah, no, absolutely.

For people listening, please do go and take a look at the project, save it for you, get it up and running it.

It doesn't take that long to get a local instance going there but Marc as well, thank you for having us and giving us an opportunity to just have a conversation about this stuff that we're really excited about. So thank you.