Ep. #30, On-Call Management with Alex Solomon of PagerDuty

Grant Miller: All right, Alex, thanks so much for joining.

Alex Solomon: Thanks for having me on.

Grant: Cool, so let's just jump right in. Tell us a little bit about your background and kind of how you got into PagerDuty and enterprise software.

Alex: Sure, sure. So I went to college for software engineering, and I've been kind of tinkering and enjoying programming from middle school.

So my parents taught me way back in the day QBasic and then I got into gaming and started wanting to make my own games. So that's how I kind of got started down that road.

Grant: Okay, cool.

Alex: And then, after college, I ended up working full time at Amazon as a software engineer.

Grant: Okay, go it.

Alex: Yeah, so at Amazon, they were one of the pioneers of DevOps. And this whole concept that as an engineer, you're writing code, you're testing it yourself.

There's no like QA team to pass it off to, you're deploying these systems into production.

And you're monitoring those 24/7, and when something breaks, you're on the hook, you're responsible for fixing the problem.

And the way Amazon did it is by building their own paging system, it was actually called being on pager duty because you carried a pager on your belt.

And when something broke, the pager would go off and you had to respond quickly. I think the response time was something like 20 minutes.

If you didn't respond, it would escalate to your boss. So you had a very strong incentive to get on and quickly.

So I carried the pager as a software engineer there and I had a few incidents that lasted like six hours, eight hours, some sleepless nights. And that wasn't necessarily fun.

But I saw the value of the DevOps way of working, I would call Amazon one of the pioneers of DevOps, one of the pioneers of you build it, you own it, this mentality.

And the way we got the inspiration for PagerDuty, it was myself and my other two co-founders, Andrew Miklas and Baskar Puvanathasan.

And we all worked at Amazon. We all carried the pager.

And after leaving Amazon, the way we came up with the PagerDuty idea was by thinking back from our Amazon days, what are interesting useful internal tools that Amazon built out of necessity that would be applicable to other companies as well.

And we thought back to the PagerDuty tool and to being on call and carrying the pager and looked around, did a bit of research didn't find any solution actually.

Tried googling and maybe spent the first month of our startup journey just doing market research and brainstorming different ideas and trying to figure out what the competitive landscape looked like.

And for the PagerDuty idea, there wasn't any solution out there, or at least not one that we could find easily on Google.

So that was kind of the origin story. And also as a kind of funny anecdote, the domain pagerduty.com was available.

If it hadn't been available. Who knows where I'd be today.

Grant: That's amazing. So you went to University of Waterloo, right? Which like, feels like that I see a lot of founders and tech folks coming out of there.

Why is that? What are they doing at Waterloo? That's like, kind of creating that type of environment that's encouraging folks to go into this ecosystem.

Alex: Yeah, sure. So I think a couple of things.

One, it's a pretty intense and rigorous engineering program, computer science, software engineering, I did software engineering myself.

And what that means is there's a lot of coursework, there's a lot of projects and you end up coding a lot.

I would say that's one big element and you work on these projects with some of your classmates where they can take weeks or even months of, sitting down and figuring out these problems and learning Linux and Unix, I think even in our first year there, we had to learn that.

We had to learn the command line and kind of get used to a non Windows environment, if you will.

And I think the other big element is the way the program is structured is you do one term of school, which is like, three months. And then you do one term of basically an internship.

Grant: Oh, like co op?

Alex: Yeah.

Grant: Oh, interesting. We had that at my university as well.

Alex: Yeah, exactly. And it's actually not even optional. It's mandatory it's part of the program. And at the end of, it takes a little bit longer.

It's not four years, you don't get your summers off. But at the end of the program, you end up with like two plus years of actual real world experience, you end up coding in a real environment you work on real software projects.

So my two last co op terms, were at Amazon, actually. And that's how I ended up there full time.

Grant: That's amazing. I actually really love co oping. It's like a modern day version of apprenticeship, right?

Because you're in school, but then you're going to a job, every four months, and you're spending that time in real work environments, solving real software problems or whatever you're doing.

Alex: Yeah.

Grant: Because then you take that same experience, you bring it back to the classroom for conversations. And it's like, it's less theoretical. And it's more practical.

Alex: Yeah and you're coming out, like I said, with the equivalent of not your first job out of school, but you already have two to three years of experience.

Grant: Of work experience, yeah. I forgot that that's one of the things that you really get, because everybody wants some, years of work experience.

And getting that as a co op is a much easier route than trying to collect that as your first job, right?

Alex: Absolutely and even better for me and in my case, and a lot of my classmates cases, as well is the last couple of co-ops, or maybe the last one.

If they like you, if you do well, they'll get a full time job. So you don't even have to go and hit the pavement and do 20 different interviews.

Grant: That's great. Okay, so let's just set the context for when this was in Amazon's life cycles, its like 2006 right?

Alex: Yes, 2006 to-- So I was there from '06 to '08.

Grant: So AWS was maybe just launching at that point.

Alex: Exactly yeah. So they launched AWS in oh seven while I was there, and I had nothing to do with it.

But it was kind of a big deal. And it came out of Amazon's experience of running these big data centers and virtualized servers.

They thought, if we know how to do this at scale, we should make product out of it.

Grant: Right.

Alex: They also launched Kindle while I was there, but I had nothing to do with that either.

Grant: But you are an avid Kindle reader, so yeah. So okay, when you were at Amazon, you were on pager duty, were you using AWS or AWS like systems to manage the services that you were? And what service were you actually working on?

Alex: Yeah, so I was on the procurement side, so a part of the kind of supply chain retail side of the business.

So basically, we had several different services where we would cut the purchase orders or PO's. In order to order goods into the warehouse.

One of the system that I worked on was the automated buying system.

So if someone bought something on the website, and it was out of stock, we had to procure it into the right warehouse from the right vendor, cheapest vendor, closest vendor.

So it was a pretty complex problem that ran in kind of a batch job in the middle of the night.

So for extra added fun, most of the pages or the incidents we had, happened around two, 3 a.m. when these big batch jobs ran, and if it didn't work, it would it would page down call.

Grant: That's like, can we just run this in a normal hour, please?

Alex: Yeah.

Grant: So you were working on one of these more, almost like an internal tool in itself? And like this batch ordering process.

Alex: Backend systems that are pretty core to the retail operations, yeah.

Grant: And so that was deployed like were you using sort of like programmable infrastructure in order to like, deploy the services?

Alex: Yeah, so it wasn't our infrastructure was not on AWS is still was very new, like brand new.

And these are systems that go back to that 90s like some of the first systems that were built at Amazon, so it was a little bit of an older kind of system.

It was built in C++ on top of an oracle database.

Grant: Okay, could you provision like new servers in sort of an elastic fashion? Was that part of it?

Alex: We could this Amazon built a slew of internal tools to do this. There was an internal tool for provisioning, it was all virtual servers, actually.

While I was there, we did a big shift from moving from bare metal hardware to virtualized hardware.

But again, not using AWS just internal tools that Amazon built.

Grant: But it likely inspired a lot of what AWS became right?

Alex: Yeah.

Grant: That's really cool. Did you sort of know that, that had happened like, okay, Amazon took these internal tools that we had and they launched AWS.

Was that sort of a mental connect that you had made?

Alex: Definitely, I saw that Amazon did that. And I thought also, the forcing function to me for starting PagerDuty was, it seemed like rightfully or not, it wasn't that hard to get a job.

And I figured, like Amazon would always be there if I needed a job. I didn't have any big commitments.

I didn't have a house, didn't have kids or anything.

So I felt like the timing was right to start something and my other two co founders were kind of in the same place in their lives at the same time.

So yeah, the inspiration definitely came from Amazon from thinking like, what are some of these really interesting internal tools that Amazon built in house, out of necessity that we were using. And we had several different ideas.

We didn't have any ideas that were directly related to AWS and AWS was like a brand new thing.

One of the other things that they built in house that we used a ton was their Facebook tool, which is kind of like the employee directory, basically.

So if you needed to know someone, someone's email, someone's contact info, their manager. It also had like, some badges on there.

Like if you want like one of the internal awards, there was one award around just innovating and trying a new project and solving a pain point and Amazon, you would get those badges on your internal Facebook.

So we considered doing that one as well. But the PagerDuty idea kind of really stuck with us 'cause I think it was one of those things where its the right place the right time.

Of course developers should be on call for their systems at their building. It was obvious to us but maybe not obvious to everyone else.

Grant: Sure.

Alex: And there's definitely a luck factor to be figured in here because the DevOps movement had just started basically, and I don't even know if the word DevOps was that popular back then.

But one of the first DevOps oriented conferences was just starting off of Velocity.

Grant: Oh, sure.

Alex: Back in oh nine, 2010. And yeah, it was right place, right time picking the right idea with a team that kind of saw the value of this way of working in that environment.

Grant: Okay, so now, you have this idea, you realize, hey, the rest of the world needs this internal tool.

We're not locked down. We don't have like, a lot of responsibility, in our personal lives.

So we can start a company, and three of you break away and start doing this. So like, I'm guessing it wasn't like instant, overnight success.

There was some challenges along the way, especially in the early days.

What were the early days, like? How did you sort of get your first customers? Walk us through those first that year or two.

Alex: Sure, sure. So yeah, I mean, we started it officially in February of 2009.

We spent January just doing research and brainstorming different ideas and trying to figure out what to work on.

Grant: Sure, maybe this Facebook thing maybe the pager--

Alex: Yeah, exactly. We had some ideas around. So but if you think back to oh nine, what were the hot ideas at the time it was like Icinga, the iOS App Store had just launched maybe like a few months or a year before that.

So there were a lot of app ideas, but nothing really resonated with us, the thematic that we picked was let's build a business where we can build a product and charge customers money.

And that was not a popular way of thinking back then. It was more about eyeballs. It was more about ad driven revenue models.

I remember Zappos was one of our inspirations because it was a simple idea it was like, product, you charge money for it.

And we never had any initial vision of building a huge company, we wanted to build maybe a bootstrap company actually where we can grow out of our revenue.

So we knew we, our domain was more from a SaaS orientation, building an online service similar to things we've seen at Amazon.

And we just need to find a real problem to tackle and to start kind of a subscription type model behind it to make money quickly.

So we started in February of '09. It took us until the summer of '09 to launch the first release of the beta of PagerDuty.

So initially, we launched it as a free service. We called it Beta and we started we just wanted customers and we wanted feedback. And we wanted to start iterating.

Grant: Sure.

Alex: And what led us to launch was Twilio was one of the providers that we used to make phone calls for--

So I'll describe what PagerDuty did early on. So it's basically an on call management and alerting service that sits on top of your monitoring tools.

And when a monitoring tool detects a problem, it sends an alert, we aggregate all of those alerts from across a variety of different tools.

Back then it was like Nagios was really popular and on prem kind of infrastructure monitoring tool.

CloudWatch didn't exist, Datadog wasn't around either. I think New Relic might have been around. So they were one of our early integrations.

But we were focused on a lot of the on prem tools and like SolarWinds, Nagios. And Icinga and a bunch of those.

And we built like a very easy way to integrate. So any sort of tool that can make an API call into our API, or even send an email, because all these tools, they end up sending email alerts.

So we thought let's make it easy to integrate with any sort of tool so that we can get those alerts and aggregate them and then we can build richer integrations later on with the various popular tools.

And then once that alert comes into PagerDuty becomes an incident and it goes to the right person and pages the right person, the on call person.

If that person doesn't answer, there's a redundancy. So there's a fallback.

So there's a primary on call, then maybe if that person doesn't answer quickly, there's a secondary and tertiary, and so on.

So the value proposition was focused around making sure that when there's a problem, it doesn't fall through the cracks. You don't learn about it from your customers, you know about it quickly, you can detect it automatically. A human looks at it, and they can resolve it quickly.

And the other value proposition which we learned about later, was more focused around the quality of life of the people who are on call.

Because we would hear this from our customers all the time our early users as well is we love PagerDuty you help us sleep better at night, which is a little bit ironic when you think that we actually wake people up in the middle of the night to tell them that their stuff is down.

So yeah, that value prop was around the creation of ownership and a process for on call, because if you think before PagerDuty, what would people do?

Well, they would all be on call all the time. So you'd page the entire team, which means that in practice, some people are going to step up and become the responsive, the heroes, if you will, and others are going to shy away from that responsibility.

And really, the load's going to fall on maybe one or two individuals. And it's not like a fair load balancing way of doing this.

And so by adding a process around it by adding an easy way to schedule your own call away to like if you need a few hours to do something and get someone to cover for you.

So that was all left up to the team to horse straight if they needed to.

And it's really about the ownership accountability and transparency of the whole process so that when someone missed their page, it was clear that they did so and nothing fell through the cracks.

Grant: And so you were users this product, previously at Amazon, so you understood the problem space pretty well.

You all had pretty defined opinions around like how this should work.

And I'm sure that first version reflected a lot of that experience. As like this domain expertise, right?

Alex: Yeah, exactly. So one of the things that I didn't mention earlier, but I should mention now is that Amazon's tool, it was built on top of Remedy, which is their ticketing system.

So if there was a high severity ticket, like a SEV one or SEV two, that's the kind of major incident and that would page the on call for SEV two there was definitely a business impact, but it was more contained business impact for SEV one, it was a major business impact.

Like, for example, the shopping cart didn't work or orders weren't getting processed, like a major thing like that, that directly impacts revenue. And that's more of an all hands on deck type situation versus just paging one person on one team.

Grant: Okay, cool.

Alex: So when we went to build this for ourselves, we really had to like it wasn't just a paging system, it actually ended up being an incident management and paging system.

So we had to build the concept of incidents 'cause we didn't have a secondary ticketing system to bolt on top of. So we built kind of an all in one.

We didn't call them tickets, but it was a kind of like a ticketing system or incident management system plus the on call management and paging element of it as well.

Grant: Okay, cool. So you had that all kind of built and ready in the summer.

Alex: Yeah, but it was a very rudimentary version, it's kind of a very much of an MVP. So that first release didn't actually have incidents.

To be very honest, it only had like, so you'd plug in something like a Nagios into PagerDuty, and then you plug it into an object on our site called an alert.

And that alert was triggered. It was either green like no problems, or it was triggered, meaning there was a problem.

But we didn't have the object model to understand incidents at that point.

We did have on call schedules and we did have the alerts and the paging and the being able to get a phone call or an SMS when a problem like that would happen.

But we quickly realized from talking to our early users and customers that.

Grant: And wait who were your early using customers?

Alex: Yeah, so it was, initially it was a few different smaller SaaS companies, more early adopters.

And I remember one of the early, early customers was AdMob, which was an ad tech company that ended up getting bought by Google for close to a billion dollars, if I remember correctly.

Grant: Yeah.

Alex: And we had a few universities as well. So I think Rutgers University was one of our early customers as well.

Grant: And they were just using it for free. This is like when you launched the beta.

Alex: Yeah, it was for free. But we had pricing posted.

And actually, interestingly, we launched the paid version of PagerDuty in December of that year, in December of '09.

And we saw an increase in an uptick in usage and adoption and obviously, now revenue.

Which is a little bit surprising, because you think, "Oh, if it's free, like won't more people use it," because it's cheap or free?"

Well, it's a matter of trust, like people and we did get questions about this, like, "How is this free? Doesn't it cost you guys, money to make a phone call or send an SMS?"

And yes, it did.

But it was a matter of trust, like folks aren't going to rely on a critical service that's free and there's no business model behind it. So actually charging money lead to a big trust factor with our customer base.

Grant: That's funny because it's like, if this service doesn't work, then I'm more screwed than I was before I trusted it. So there's a lot of trust they have to have.

Alex: Exactly and becomes mission critical pretty quickly 'cause you're relying on your major incidents and the ones that need people attention.

And there could be revenue at stake. In fact, in a lot of these customers, are depending on the application that they're plugging into PagerDuty, there's definitely revenue at stake.

Think like an e-commerce application, you can actually measure your revenue loss in dollars per minute, potentially.

Grant: Right, right. And so you're part of Y Combinator. Is that when you joined, were you funded by them? Did you have other like funding at that point?

Alex: Yeah, so initially, we started the company in February of '09 in Toronto, and the following year, so...

Grant: You were working for Amazon in Toronto, or based in Seattle?

Alex: No, we were in Seattle. But we-- so all three founders are from Toronto, and after Amazon, we all moved back actually, we all moved back in with our parents to kind of reduce the burn rate.

Grant: Naturally.

Alex: Yeah and then we started the company in Toronto. And then the way we ended up in YC was by applying to YC the following year.

But at that point, we already had revenue. We already had paid customers, we had like, maybe a handful of YC companies as customers.

So we leveraged that, our traction and our customers who are already YC companies to get recommendations and get into YC.

Grant: Oh, interesting. Okay, so you were still in Toronto at that point.

Alex: We were and then we got into just summer...

Grant: Was it just the three of you at that point?

Alex: Yep, just three of us. We were maybe 10 to $20,000 in revenue per month.

Grant: Oh, wow.

Alex: Monthly revenue at that point. And I called it pizza profitable, like ramen profitable, but, maybe a little nicer, yeah.

And so we were able to pay our, at least the cost of running the product on we actually launched on AWS. So the cost of running the system it was not a big operation back then obviously.

We got into Y Combinator for the summer 2010 batch. And that's when we made the move to California to Mountain View.

Grant: Okay and so over the course of just over a year you bootstrap basically from zero to 10 to 20K in MRR.

Pretty solid, how were you getting customers early on just like from word of mouth? Like what was the primary customer acquisition?

Alex: Yeah, good question. So initially, and I meant to mention this earlier, but so we were using Twilio as one of our providers to make phone calls.

And Twilio was running a contest every I believe every month they had a different theme for applications built on top of Twilio.

And one of the months was monitoring and alerting, which seemed a perfect fit for us.

So we rushed to release our first that first beta that I talked about, and we ended up winning the Twilio contest.

And so they blogged about us and they wrote a whole blog article about PagerDuty and what we did.

And that's how we ended up getting our first set of beta customers or beta users.

And then from there, it was like slow going at first, when we launched the paid version that helped, because we pretty quickly like that month got to over $1,000 in MRR.

And then from there, it was a lot of word of mouth, really. So the DevOps community was starting to really gain critical mass.

There were a few conferences that started popping off. So I mentioned Velocity O'Reilly Velocity conference, which was run in Santa Clara back then and-

Grant: What, were you like going to these conferences, were you on stage? Were you just attending or have a booth? What was your involved?

Alex: Yeah, so one of the things that we saw instant growth from was we sponsored the Velocity conference early on, and I believe this was back in 2011.

So it's a little bit later, it's after YC. But we sponsored the conference, we had a booth, and we sent like basically half the company including myself there to talk to potential customers.

Grant: Naturally.

Alex: And as it has happened. This is another lucky moment.

One of our early customers had a keynote and they mentioned he mentioned PagerDuty in the keynote.

So that ended up drawing a lot of attention to us and we got our booth got mobbed by people wanting to learn more about PagerDuty.

Grant: Yeah.

Alex: And we in our metrics later on later on that month or next month, we saw a big spike in sign ups and paid activations of PagerDuty.

So we saw that word of mouth was like the biggest driver, we were solving a hair on fire problem that a lot of companies had, they were starting to put people on call for the first time, and even in--

So as early adopters were adopting DevOps on calls is a big part of that.

And even in larger companies like more traditional companies with the centralized it function, PagerDuty also served the function era of automation.

Instead of, picking up the phone and trying to call someone it was all kind of handled through PagerDuty and the paging and on call aspects were all handled through our system.

Grant: Okay, so this kind of reminds you what was probably happening before PagerDuty at bigger companies was that they would have like a NOC, right?

Network Operations Center, right.

And those, that would be a 24 by seven operation, where they're like monitors on all the walls and are looking at all the output and if there's an issue with some application or service, then they have a runbook or a playbook or something they would go through, try to solve it.

And then if not, they would pick up the phone and call whoever they thought should be calling at that point, maybe a VP of engineering who would then call somebody else does that seem?

Alex: Yeah, that's exactly right.

So what we were seeing is, our early adopters were more smaller companies who were trying to adopt the DevOps.

Kind of like the first wave of Cloud Native DevOps oriented companies that were basically adopting the mantra of if you're an engineer, and you're building a system or SaaS product, there was no ops team to hand it off to.

You were the person who's responsible for this and operating this and if something broke, you had to answer the page. And hence PagerDuty.

Grant: And Cloud is probably a big driver of this because those traditional NOCs were actually managing, physical data centers, right?

Alex: Yeah.

Grant: And they would have real machines. And so you kind of had to have 24 by seven on staff, like IT admins to manage those machines.

Now, everything goes to the cloud, anyone can spin up these services, in AWS.

And you didn't really, need the same level of constant attention to make sure that the machines were up and running.

And they were doing that, but now it became like the application level, the service level side, and you would have Nagios or something monitoring it.

But now you needed the way to actually like, someone instead of picking up the phone and from the NOC, PagerDuty steps in to take over.

Alex: Exactly, so that's kind of the flip side is the more traditional environments that had a network operation center, which was a 24 by seven.

Kind of like a big room with lots of screens. Think like the NASA command center that you may have seen in Apollo 13. That kind of thing.

They had a 24/7, three different shifts each eight hours. And when something broke, the folks in the NOC were there, they were not on call, they were just kind of doing their regular shift.

And when something would break, it would pop up on one of the screens, they would see something flashing red, they would investigate it.

They had a runbook, which basically means, oh, if this system is having this problem, here's what to do first here, if that doesn't work, here's what to do second, third, and so on.

It's as basic as step by step sequence that you would follow.

But generally speaking, what we learned over time is that the folks who are in these NOCs are not necessarily the most senior folks. They're actually the lowest end of the totem poles kind of more of an entry level position. You're not necessarily an engineer or coder. You don't necessarily have the depth of knowledge on how to fix a lot of these problems. You can maybe fix some of the simple ones that are routine. But then if they are simple and routine, can't they be automated?

That's the question. So we started getting into some of these environments as well, not so much from the point of view of, these folks are not on call.

But if they encountered a problem and they didn't know how to solve it, if there's a more complex problem, if it was a major incident that was impacting lots of different systems, anything that's more unique and not routine, then they would have to escalated it and loop in other subject matter experts.

And also loop in leadership and maybe some of their VPS or C level type folks to draw attention to the problem as well.

So PagerDuty would come into those environments as well through the value out of automation.

So instead of running on calls, scheduling a spreadsheet, and it's all manual, and you pick up the phone and you dial someone, and if they don't answer you have to call them again or call someone else.

Just create the incident to PagerDuty and then it takes care of the rest and loops in the right person.

And again, remember nothing falls through the cracks. So it will keep bugging you until you pick up or someone does.

Grant: Rewinding again, right so back to this sort of initial product.

So one of the things that people always talk about with like PagerDuty is like, how is paging people like, this big problem? How is it a publicly traded company, right?

I mean, I'm sure you face that your entire time that the TAM, the total addressable market just isn't big enough.

So who saw the vision? Who understood the trend lines are all going in the right direction.

And I mean, correct me if I'm wrong, but I'm guessing that is the thing that people have said to you for a long time, right?

Alex: Yeah, a little bit. I mean, I think part of the problem is the name of the company is PagerDuty, which, if you're not from this space, if you're not coming out of more of a technical background, you may not, you'd think oh, it's just paging?

Isn't that a solved problem? But yes, there's positives and negatives to the name it puts us a little bit in a box, oh, it's just an alerting and paging system.

Well no, the name of the company is PagerDuty, but yes, we started as a paging and alerting system.

Grant: I mean Salesforce does everything.

Alex: Exactly.

Grant: It's fine, the name is great, but generally, I'm guessing that many VCs, specifically, I say this because, I hear this all the time, about my company, right, like, what's the TAM, everything else.

So you had to have heard that. But somebody got it, somebody was like, this is a huge trend. So how did they get it? And how did you combat that kind of conversation?

Alex: Yeah, so I don't think we had too many questions about the number of potential customers because we started by writing the DevOps trend.

And yes, that was fairly new back in oh nine, 2010, 2011, where we were raising money, but it was also the trend of software eating the world.

I mean, you've heard that phrase over and over. So yes, there are software companies and those were early adopters, but every company beyond a certain size has to become a software company has to innovate.

And if you look at transportation, if you look at retail, all of these industries are becoming software centric, even ones like oil and gas.

Well, if you automate things if you leverage software to become more competitive, I mean, it's clear that this is a huge trend.

And nowadays, you see folks who are on this C level suite of these fortune 500 companies that have the title of Chief Digital Officer, so the digital transformation of the world was a trend that VCs understood fairly well.

Grant: I totally agree. And I think hindsight, huge market opportunities, obviously in the world.

But software eating the world wasn't a thing in 2011. That's when it like, came out, right?

So like, maybe people just didn't tell you but I feel like people had to feel like this was--

Because I've heard the same thing about companies like LaunchDarkly right, like feature flags of service, how's that a company?

It's great company, it's a huge company. They're doing incredibly well. They'll be on the same, PagerDuty track as well to Typio someday.

And I guess my thought is like, there's this really interesting piece where you started, focused and like that tip of the spear became, you found a huge opportunity with this sort of growing market and finding this pain point.

So who did your seed round 'cause that feels like the people that are really solid?

Alex: Yeah, so yeah, I mean the initial money came from Y Combinator and it wasn't a lot.

It was like 20,000. So back then YC didn't put a lot of money into the company, we didn't do YC for the money per se.

I mean, 20K was fine. But it was more for the network.

It was for the help that they were able to provide it was for being able to reach a bunch of companies that are connected to YC or that are former YC companies as customers.

And then the biggest value add for YC for us was the, a couple of things.

One is them, we work directly with Paul Graham as him helping us formulate our larger vision because we did start focused on call management and alerting and he kind of pushed us to think bigger and to kind of, think how are you going to become really big as something that a VC would be interested in investing in?

And then two, it's getting connected to the VCs and understanding how they think and what they care about and what kind of questions they care about.

Basically, like, remember, PagerDuty was started by three engineers.

None of us had done any kind of business oriented, none of us were CEOs before CTOs before, we were all coders. So learning the side of the business of fundraising was really important.

And that's where I feel like YC helped the most.

First day they introduced us to a whole bunch of angels and investors and part of the program was connecting us to investors to mentor us. But it was kind of like a two pronged approach. One is to get advice from them, but obviously, if they like you enough, then they'll actually want to invest as well.

So we got paired up with Michael Dearing, and we got paired up with James Lindenbaum was CEO of Heroku and one the founders of Heroku, and they both helped us quite a bit.

And James actually helped us quite a bit with investors as well. And with thinking of the bigger pitch, the bigger vision for PagerDuty.

And we iterated on that and came up with a larger vision and back then I remember it pretty clearly, like step one was yes on call measurement alerting.

Step two was to expand to become a full incident management system end to end so that includes everything from you have all of these alerts and events.

How do you reduce noise? How do you group related things together?

How do you maybe apply things like machine learning to understand that the data set that you're dealing with to understand the blast radius of the problem that you're dealing with, and be able to maybe even predict some problems before they become like a huge outage.

And then finally, along the same vein is around having analytics and understanding and learning from your incidents and problems over time, and then step three of the vision was to take on the big ticketing players.

So back then Remedy was kind of the dominant force. And over time, as we learned, ServiceNow kind of became the de facto standard. So that was our vision back then.

And it's interesting, because I've had folks ask me like, "Was that your vision back then? "How, how much has it changed?" And I would say directionally, it was pretty spot on.

Yeah. Like, okay, we're not taking on Remedy or ServiceNow directly. But we've been able to over the last 10, 11 years now build a end to end incident management system.

We've gone into the AI op space and built an event management solution.

Our product there is called PagerDuty Event Intelligence. And it has the machine learning elements that I talked about so that you can predict outages before they impact your customers.

And like I said, our vision, a lot of the elements that we came up with back in, 2010, 2011 are still true today.

So I wanted to answer your question earlier about how do we acquire customers. It was a lot of word of mouth.

It was the emergence of the of the DevOps community, where folks were very vocal.

They were on social media, they're talking about interesting tools and practices and processes that they learned about. And PagerDuty came up in those conversations a lot.

So what would happen is folks who would mention us at conferences, they would talk about their tool chain.

And PagerDuty was one of the critical tools in there because it enabled the concept of full service ownership or putting developers on call.

So that was pretty core. And then folks who keep talking about us and speaking about us at conferences, and people would change companies and they would adopt PagerDuty at their first company.

And then they would search a second company and leave PagerDuty behind an adopt PagerDuty, at their second company and that groundswell just kept growing and growing.

And I also remember, we were trying all kinds of things to acquire customers faster, like Google AdWords, display ads, but none of that stuff really worked.

It was really coming down to what I mentioned before, which is solving a hair on fire problem, riding a kind of a wave for us, it was the trend of the DevOps methodology spreading everywhere.

And it was that word of mouth element in that community.

Grant: Yeah, that's really interesting.

I think you kind of mentioned something there about putting developers on call or putting developers in this sort of, you said it more eloquently than I just repeated it.

But like, did you like write about that? Or did that just, did other people just like kind of present about that, and then include you as evidence of that as a thing.

Alex: We did some writing about it too. And, like, we looked at what other companies had done and content marketing was one of these longer term strategies that we saw the value of.

Like blogging and talking and speaking at conferences and doing this in such a way that it's not shilling for the company or for PagerDuty, but genuinely caring about a better way to do things.

Grant: Yeah.

Alex: And we did this but what helped us even more was our customers doing this because it's always better to hear from the horse's mouth than from the vendor that has an incentive to sell you something.

Grant: Yeah, and then, so your customers are out there talking about their overall move to DevOps, everything else and then being not necessarily.

And we do PagerDuty, everything right? Instead of its like, oh, yeah, we're doing all these things.

And PagerDuty is an important part of this whole process. It's actually better because you're like, it feels less like an infomercial and more like a--

here's the tool chain you should use in order to make a similar transition. That's cool.

Alex: Yeah.

Grant: So now you're kind of launched, you're out there. When did you like, become an enterprise software company?

Because I'm guessing when you launched it, you didn't really feel like it was an enterprise software product. It felt more like a standard SaaS service, right?

Alex: Yeah, so I'll tell you some of the thinking back then. So, remembering that PagerDuty was started by three engineers.

We looked around at other software companies what we really loved about companies like say I'll use an example of 37signals now renamed to Basecamp, but their whole mantra was around, sell software online, make it easy to buy and adopt.

Star with a credit card, make it like self service, because that's what people want to buy. And we really believed in that. And that's exactly what we did.

And that's how we launched it was a free trial available online, anybody could sign up, you didn't even need a credit card to start a trial.

But at the end of the trial, which was initially 30 days, you would have to put in your credit card to keep going and keep using the product.

And initially, we didn't, I was not a believer in needing salespeople because I thought well, who would use this engineers, I'm an engineer, how do I buy stuff?

Well, I look on the website I learn about the product, I start a free trial. I don't like talking to someone, I can figure it out myself.

I like to kind of tinker and try it out. And if the product works as advertised, then yeah, put in a credit card and you're good to go.

And if you need to call a salesperson then it's clearly too expensive for me to even try. So that was kind of my mentality back then.

And of course resonated with my other co-founders as well. And what we found in practice is that, yes, that did work. And it worked really well.

And companies would start with a free trial, they would start adopting.

And then it was very much a land and expand model where they would start with maybe one team of people.

And then they proved the value. And then it was spread to another team and another team, maybe a whole department and then maybe multiple departments.

So early on, some of our first customers were more SaaS startups, and then we started getting larger companies as well.

Like I remember, Nike was one of our early customers. And they started with three users on a credit card then in the course of several months they expanded to hundreds of users.

And at one point they were running like over $10,000 a month on a credit card, which is pretty substantial amount without having talked to anybody at PagerDuty, which is the crazy part.

Grant: Do you know, was that like someone's personal card or was that like a corporate card?

Alex: I'm pretty sure was a corporate card.

Grant: I mean, it's a lot of miles, to end up to that.

Alex: But then what was interesting about that journey is that yes, that worked really well.

But we started getting more and more inbound inquiries of folks, our customers wanting to talk to us to ask us questions about the product, about the infrastructure about the resilience and reliability because it's so important for a tool that.

Grant: That trust aspect.

Alex: Yeah, that trust aspect and that, the business we're in, which is we need to be up when you're down, and we need to be pretty reliable.

We're like, we build ourselves that first version of the product as the 911 for IT, so you can't call 911 and then it doesn't work. It has to work every single time.

So it was that trust element of them understanding can we really trust, is this a real company are they going to be around next year?

So all those questions and also there's like the kind of regular part of getting a deal done which is getting a contract done.

Now we had the on the online terms and you we thought maybe everyone's going to be fine with that.

But it turns out they weren't. They wanted real contracts signed.

And what ended up happening is I was spending more and more of my time on the sales side of things, helping our customers through this process, doing contracts, negotiating that working with the lawyers.

And then I realized well, seems like there's a need here like the sales thing it seems like we need to hire some people here because I have other things to do as well, like hiring and managing and setting priorities.

Grant: You're like, "I've already done this..."

Alex: Fundraising and all those-

Grant: I've already done this exact same contract negotiation three times and somebody else can do it now.

Alex: Exactly, exactly. So we ended up hiring our first two salespeople, I want to say back in 2012.

And what was interesting about them is they're both coming out of an engineering background, but then had switched careers into sales.

And there were folks that I had kind of gotten to know, one started with us as an intern. And then midway through his internship co-op actually from Waterloo.

Grant: Oh, wow.

Alex: He wasn't doing an engineering co-op.

He was doing more of like started out doing market research and researching customers and figuring out who should we talk to these potential customers really.

And then one day he asked me "Can I do sales?" So I asked him, "What do you know about sales? Have you done it before?"

And he said, "Yeah, actually I did at Salesforce, Zendesk, and this is what I did. And these are the customers I worked with."

And I was like, "Wow, that's pretty cool. "Let's give it a try."

And the other salesperson that joined about the same time he came out of a different software company.

He, his background was he was an engineer at Splunk years ago. And then he went into sales.

So yeah, they were both technical, we didn't need to hire a sales engineer. To help and do demos.

They already knew the technology they knew pretty in depth. And they had the kind of customer relationship side of things as well. So after we hired them, and I believe they both started around the same time, around the same month in 2012. We saw a big spike in revenue. So it clearly worked and it was a good addition to that self service model because we were getting inbound questions.

And then we can go talk to some of our larger customers, the ones that are growing the fastest and talk to them and figure out what's going on over there.

How can we help you? Can we help you deploy faster? Can we do a larger deal? If we had to do these contracts, so they could deal with the heavy lifting of that as well.

So it worked out really well. And we got one of them to specialize around the larger customer.

So he was the enterprise sales guy, and the other one was more everything else. So mid market and below.

And this all started by you asking, when did we realize we were an enterprise company?

Well, because our models started with online self serve.

But then pretty early on, we started getting some large customers and those customers we started talking to our customers, certainly from a product development perspective, like we wanted to understand.

Okay, what are your challenges? What else can we build for you? What are your pain points? What is PagerDuty, doing really well for you? And what what's missing?

What features are missing, what should we be working on next? We started to understand that they had a lot of needs.

And some of the some of the early questions were around like simple things like permissions and being able to organize larger account, lots of folks into teams.

Like, can you add a element of a team so I can organize my people? I don't want 100 people all in the same team, if you will.

I want to be able to organize them into teams so that you can only see what you care about in the product. Not everything.

Grant: Sure, okay. That's actually really interesting.

So the first sort of some that first off outside of contracting was really some of these enterprise feature requests, right?

And so it sounds like role based access control was sort of that, first one in terms of, hey, let's put some teams in that's organizing group people.

I'm guessing if you had hundreds of folks, some kind of single sign on, maybe or not?

Alex: Single sign on. Yeah, absolutely. That was becoming pretty big back in kind of, 2013, 2014.

Grant: Yeah, it was kind of just getting really kind of more popular at that point, I guess, right?

Okay, so as you're getting these requests, were you processing them all pretty well being like, okay, these all sound similar, right?

There's a common thread here, and these are generally the EnterpriseReady features right? These are RBAC this is SSO.

Did you end up doing some kind of like, reporting stuff, was that a feature?

Alex: Exactly, yeah. So exactly right.

Like, customers were asking, "Okay, I have all these incidents and alerts and folks on call and on PagerDuty. Can I see, the number of incidents? Is the volume getting better or worse over time?

"Can I see it from a service perspective? Like, I'm running these online services or these micro services? Is the health of the service getting better over time, or worse over time?"

"What about the on call pain? Are people getting woken up all the time? Or is it getting better or worse?"

So they wanted to see that at an individual level, team level, org level, service level, like all of these were questions that our customers were asking and another way that we discovered some of these needs was, we took the approach of having an open API for all the functionality that you could do.

If you had the UI, like anything you could do via the UI, you can do via the API. And we made that decision early on from maybe our first or second year being in business. And it worked out really well.

Because as I think everyone knows, now, API's are really important.

Especially as you have a larger deployment, maybe hundreds of users, maybe thousands of users on PagerDuty, or any other SaaS tool, you're not going to want to necessarily, use the UI to add every single user, add every single team, add every single service schedule, et cetera.

You'd want to automate some of that via the API. So the API access has become important.

And you mentioned reporting actually, what we saw with some of our early customers.

I think Heroku, if I remember correctly did this is they leveraged our API to build their own reporting system on top of it.

So they're able to pull the data out of PagerDuty, and then run reports and graphs and see our people generally happier, or are they getting paged 500 times at 4 a.m.?

So we saw our customers building this stuff on top of our API. So it was very clear to us. Oh, shit, we need a reporting module. So let's build that.

Grant: That's great. Okay, and then did you make all these features available to every customer?

Or did you do some amount of, sort of good, better, best, product assortment?

Alex: Yeah, so we did that, exactly. So we started out with maybe a single package or a single plan.

And then over time, we ended up with three, kind of a starter version, middle of the road version, enterprise version, which has all the bells and whistles and over time we, every new feature that we added we had to consider, okay, do we want to make this available for everyone?

Or should we make this available on one of the higher plans?

The field of pricing is pretty it's a whole domain of...

Grant: Yeah, interestingly-

Alex: Pretty deep domain.

Grant: The concept of product assortment on the EnterpriseReady guide actually came from a talk that Michael Dearing from Harrison Metal gave at Heavybit, that James started.

And that talk sort of helped inform this idea. Like it was so important to have good, better, best.

And ultimately, a lot of the features that we extracted sort of identified as enterprise features, were just from us surveying landscape of pricing pages and saying, what falls into the enterprise category for these companies, right?

Alex: Yep.

Grant: And ultimately, it's a very common set of features, right? So, PagerDuty is a perfect case study of all of these pieces, so.

Alex: Yeah, absolutely.

Grant: Okay and so like those first enterprise customers, did everybody still come in inbound and you were basically upselling them over time?

Or did you start to do some outbound and get some customers with bigger contracts starting earlier?

Alex: The first wave of customers definitely started inbound. And that inbound engine kept going. And it's still going today.

Like we still have an online self service model where folks can start with a free trial. The trial is 14 days now.

And you can kick the tires try out the full featured product, but we don't limit anything on the free trial.

So you can kind of take advantage of everything and see how everything works. And we still get lots of folks coming in via that channel.

Generally speaking, it would be someone from the technical side, maybe a team lead or a manager trying out PagerDuty.

Going to a bit of the sales evolution of the company. I mentioned that we started out with no sales.

And then we added our two first salespeople, and they were inside sales. So they mostly on the phone, email, kind of the usual inside sales channels.

And over time, we added this other tier of sales called a hybrid.

So it's kind of a mix of field and inside that first hybrid team was all based in San Francisco where we're headquartered, and they were mostly on the phone but from time to time for the right customer.

For larger customers, larger enterprise customers, they would actually get on a plane and go meet them face to face because there's an element to sales that's all around relationship building.

And a lot of companies buy that way, a s they need someone to talk to they want to see someone face to face so that they can really build that relationship and that trust with the company.

Grant: Sales is still a human to human thing, right? So software is not truly a commodity.

And so there's a lot of, sort of nuance to it. And trust is a big part of that. And so like you build trust by meeting people, and it makes sense to go there in person.

I'm sure you met a lot of customers in person and in prospection person to help seal that trust.

Alex: Yeah, I would get called in on some of the larger deals. One of the big benefits of being based in San Francisco is because of a lot of our early adopters were software companies.

I mean, that's the place to be and it was so easy to just walk down the street and meet with Heroku and then walk a little bit further and meet with the next customer and it was great.

I remember Apple was one of our early customers as well and back in like 2011, 2012. We went down to Cupertino.

I had a few meetings with some of the folks there to understand what they were building and they were going online in a big way with iTunes. And some of their online services.

So it a big driver for changing the way they work and getting us into the automation side of their operation center, their 24 by seven operation.

And then over time, they started moving into more of a service orientation.

I mean, now, if you look at Apple's quarterly report, it's all about building a services business, not just a hardware business, and we play a pretty big role in that.

Grant: That's pretty cool. And, I mean, it sounds like there was a pretty constant drumbeat towards growth and towards a lot of companies have some ups and downs.

I'm sure there's some parts but did it ever feel like this company might not work?

Was there ever a moment of, "oh my gosh, this is all going to go fall apart, in a week" or anything?

Alex: Well, I mean, there were a few cases where we had some stuff go wrong, like in the first year after we launched our free beta.

We started getting some early customers but then for a few months there nothing really happened.

We had maybe like a dozen early signups and folks, some of those were using the products. Some of them were just kind of signed up and then abandoned.

And then it was a few months of nothing nobody was signing up.

But that's why we were trying like AdWords and display ads and some of those things to promote the product and get the word out there that, that PagerDuty exists.

But and this is before the word of mouth really started rolling.

So there were a few months there where we felt like, well, it's not working, but we haven't given it enough time.

Should we pivot? There were some internal conversations around that. But we stuck with it.

And it started taking off, obviously, slowly at first, it was exponential growth, but it was exponential off of a very small base.

Grant: Sure, sure. Okay, cool. But I mean, since then, still pretty much, just up into the right sales keep going.

Things are going I mean, you IPOed what, last year or something or?

Alex: Yeah, almost a year ago last April.

Grabt: Amazing it's a very fortunate position to be in for that to go so continuously well, so congratulations. I'm sure you've worked really hard.

Alex: Thank you.

Grant: So let's talk a little bit about, throughout that process, at some point, you were the CEO for many years, right? For the first how many seven? For the first seven years yeah. Okay but then you hired a CEO.

Alex: Yeah.

Grant: So talk us through that. How, why, who the whole thing, what made you decide that was the right thing to do?

Alex: Yeah, yeah. So at that point, the company had gotten to a fairly significant scale.

We were over 150 people. We're getting close to $50 million dollars in annual recurring revenue.

So we weren't like a small kind of pre product market fit. It was clear we had product market fit, it was clear we had traction.

We had done a couple of rounds of big rounds of funding the series A and Series B.

And I was looking at the product side and I was finding that I didn't have much time, because I was, hiring executives, managing, setting goals.

I wasn't spending very much time at all in the field, talking to customers, thinking about the product strategy, I just didn't have the bandwidth for that.

And I also felt like, we were innovating, but maybe we could be innovating even faster.

So I was kind of really, I wanted to focus more on how do we improve our velocity to innovate even faster? And just didn't have the bandwidth to do so.

So that's what initially started that process and that ball rolling.

And I initially thought, well, maybe I can hire a CEO, like a chief operating officer, someone who can manage basically everything except for product and engineering.

Grant: Sure.

Alex: And started down that path of doing some research and talking to people I talked to folks like James Lindenbaum, and asked for advice.

And one of the things I kept hearing over and over is that it's very hard to hire a CEO that's strategic enough and can really own the go-to-market and that the people who are amazing at that are going to command the CEO title.

They're not going to want to work for a CEO, but kind of in this secondary role, if you will.

A nd I had met founders who had tried doing the same thing, and after a year, they just hadn't found the right person and then switch tracks to replace themselves.

And then there were, other cases where founders had tried to hire a COO and they did hire a COO but found that they solved some problems with the hire but created a slew of new problems and leading to that not strategic enough that I had me on earlier.

So I didn't see any cases really where it worked out that you could find a very strategic COO who didn't want that CEO title.

So then I decided to with the support of the board, just switch tracks and hire a CEO and replace myself.

And I mean, it's a huge decision, obviously, because, you're giving up control, you're basically trusting someone else to run the business and you would then work for that person.

So that was, I mean, the joke I like to make all the time is this is a bigger decision than getting married because if that's doesn't work, you only lose half your stuff.

So I mean, this is all or nothing.

Grant: Yeah.

Alex: So we, with the help of the board. And Andreessen was very involved in this.

We have John O'Farrell from Andreessen on our board. And they have, one of their recruiters is very, very strategic.

And he, Jeff Stump, he helped us a lot in the search, like really helped us kind of put together the criteria: what are the must haves, the nice to haves, what are the goals that this person would have for the first year in the job?

And then he helped us design a set of interview questions.

And then part of this process is also talking to other founders, and other professional CEOs to learn. Okay, how do I hire one of you?

Grant: Yeah.

Alex: And how do I evaluate to make sure that I'm getting what's best for the company.

And what we learned through that process is that the partnership between myself and this new person was really important.

The cultural fit was really important, like making sure that they fit into the culture that they don't come in and maybe they're a great operator and a great CEO, but then they blow everything up and maybe fire the entire executive team.

And we have to start over. So those are really important factors as part of that decision.

And then we started the process, we hired a search firm, because again, this is a big time, high stakes decision.

Grant: Yeah.

Alex: And you have to get the best people out there and get access to the best people out there.

And it was not a hugely long process, actually, it ended up being like three or four months from the time we hired the search firm, to the time we made an offer.

And we hired Jennifer Tejada. She's been fantastic. She's been at PagerDuty now for three years, over three years.

And she's helped us go public last year, and it's been great.

Grant: And where was she before?

Alex: So she came out of a software background and then before that out of Procter and Gamble, like way back, but she was at Keynote Systems, which is kind of in our space, infrastructure monitoring.

And then she was at a software company before that in the mining space like natural resource management space.

Grant: Interesting.

Alex: So she had and what I liked about her a lot is that the cultural fit was there.

And that she was one of these maybe unique individuals, you can call her a unicorn and that she's really good at a lot of different things.

Whereas a lot of the other candidates that we talked to is they have their specialty that they're good in one area, like marketing or sales, or these, but they're more T shaped they're more generalists who have expertise in one area.

And what Jennifer brought to the table is she had depth in multiple areas, she had a marketing background, she had a product background.

She had run sales before and carried a bag. So it's like kind of a unique unicorn type of individual.

Grant: Yeah, okay, that's interesting.

Alex: And then lots of reference checks. That was the other part of it, especially, back channel reference checks.

Where it's not one of the people that she would give herself. But what was interesting is she basically opened up her whole Rolodex and said, "Call anyone."

And I ended up calling I remember through the process I called someone who she had fired in her past and that person was still very positive.

Grant: Really.

Alex: So that was like, man, you never hear that someone who was fired by Jennifer and still gave her a raving review.

Grant: Interesting.

Alex: I can't imagine that if you called everyone in my past that you would get all perfect reviews.

Grant: Sure and me too. We've rubbed some people the wrong way. And we were young and we did the wrong way and you're like, "Oh man maybe I should apologize for that."

But when you called, were they actually everything was totally positive or were there some that were like you had to just like weigh it a little bit and be like, "Okay, I kind of understand. Maybe this person's a little lukewarm."

Alex: No, it was all positive I've never done reference checks.

Well, first of all, we were very so it was myself and John who was on the board. We were on the same calls together for a lot of the reference checks and he said the same thing.

He's done tones of reference checks in his background. And we've never, either of us have seen such positive reviews with any other candidate.

Grant: Man, I feel bad about myself now. Sorry for all those people who have wronged.

I'll have to call up and atone so I can get positive reference checks throughout my career. Starting here.

Okay, so that's amazing. And she's, I'm guessing, bringing her on just took so many things off your plate you've just felt like, just I'm guessing like a weight was lifted off your shoulders.

Probably did spend a lot of time onboarding or getting her into the business and all that.

But at some point, I'm sure it just started to feel like you were being, elevated and she was taking over is that right?

Alex: Exactly, yeah so I spent a lot of time up front, just basically downloading everything I knew and helping her on board and getting her all the strategy decks all the fundraising decks, all the kind of materials that we've put together, board decks that we presented in our board meetings, all of that, so that she had the context, as she knew.

And part of what I think helped is part of the process is I was very transparent with what was going really well.

A lot of things were going really well we were growing faster, exponential growth, creating a new category.

So a lot of things are very exciting. But there were things that weren't perfect either.

And I don't think any company is perfect but I was very upfront and transparent with the things that weren't going well.

Grant: Yeah.

Alex: And I think that was maybe a refreshing breath of air and it's part of our culture as well.

We do want to talk about the problems and not kid ourselves that everything's all sunshine and rainbows.

So that was part of the interview process and then when she joined as well so that there weren't any major shockers or surprises or bait and switch kind of things like she thought she was walking into a perfect situation and then things weren't so perfect once you, once you open up the kimono, but--

Grant: I think that's so important and I do it with any like somewhat strategic hire like try to make sure they understand all the skeletons in the closet, anything else like hey, here's what's going great, h ere's what the challenges are.

And the candidates actually really appreciate it. And I think it also just once they start it creates a much like there's a lot more trust.

Alex: Absolutely, absolutely. So yeah, I spent a lot of time on boarding. And then it was really about the relationship and just communicating a lot and making sure.

One of the pieces of advice that I got, before hiring Jennifer was around, making sure that we're aligned.

And now, I'm giving up my position as CEO, I'm going to work for her, but I'm also on the board. So in a small way, I'm also her boss.

Grant: Yep.

Alex: And it's important not to undermine or not to, undermine publicly or make it seem to the rest of the team or to the exec team, that there's a conflict there.

Grant: Yeah.

Alex: So I was very attentive to that to make sure that didn't happen.

Grant: And I also love that idea, Mark, my co founder or CTO, I'm the CEO.

So I'm his boss, on that paper, but we made him the chairman of the board. So he's also my boss in that regard.

So it's sort of self referencing in that same way, it just kind of balances. There's no power tripping. It's we all work for each other realistically.

Alex: Yeah.

Grant: And so you kind of mentioned openness and I know that you guys have published a bunch of interesting things I you think you published your security training deck.

Alex: Yeah.

Grant: And some other pieces, that I think are really interesting things to share, which I love.

So first, what are the different things you have shared? Or some of that top, sort of internal things you've shared? And then kind of why?

Alex: Yeah, it's a great question.

So we started down the path, a little bit organically of publishing some of our internal best practices, because we thought well, okay, so we have an incident management and AI ops product, and that's our space, and that's our category.

So we were pretty good at responding to incidents ourselves internally.

We documented this process and we adopted elements from emergency response, if you think firefighters or other first responders respond to incidents, some of those elements we borrowed from that process.

Some of it came from our own past background, so like at Amazon and other companies who are pioneers in this space.

And we wrote down this process and then one of our engineers ended publishing it in an open source way.

So it's available on our site, it's actually response.pagerduty.com and so our incident response practice, the one that we use internally at PagerDuty and it talks about the things that you must do to prepare for an incident, you should practice this process if you don't have a lot of incidents its still a muscle that you have.

And that, that is being exercised. And it's important to define the roles upfront of all the folks that are in the incident process.

So I'm not going to go into too much detail there but check it out response.pagerduty.com. We call this a library of ops guides. So that was our first ops guide.

And then over time, we created one on post mortems and why doing blameless post mortems is really important and how to conduct a post mortem analysis and investigation.

We did one on operational reviews, what are the metrics that matter when you're looking at your operations and your service health and your team health?

And what should you review on a weekly monthly and quarterly basis.

Grant: Like SLO, SLA?

Alex: Yeah, that's in there as well. And then we just most recently did one on full service ownership.

So this is like a broader concept around the whole DevOps and how to manage services and how to put a new service think like a micro service or application into production.

And what should you care about from the things that you should look at monitoring, alerting, incident management, continuous deployment, testing all those things.

And then how do you a retire service? You're not going to keep everything running forever. But yes, basically, we created a library of these guides, and those are available at pagerduty.com/ops-guides.

And as part of that, we also started releasing some of our training that's not directly applicable to say post mortems or incident response or incident management.

So one of the ones that we published is our security training, which that first engineer that published the original ops guide on incident response, then switched teams and went from SRE into our security team.

Grant: Oh, funny.

Alex: And he developed a set of trainings for internal use. So one is for engineers, which is more focused towards the folks who have GitHub access and are committing code.

And what to think about to protect PagerDuty from getting breached or anything along those lines.

And then we also have a set of security trainings around for all employees of PagerDuty, which talks about say things like password management best practices and making sure that, you're aware of phishing attacks and things like that.

We've had a case where, back when I was CEO when the CFO at the time got an email that looked like it came from me to transfer some amount of money to an account.

Grant: Spearfishing.

Alex: Spearfishing and very targeted approaches where it looked like it came from me because if you're not paying attention, it looked like it was Alex Solomon from PagerDuty but if you kind of dug in a little bit, you could see that it was a phishing attack.

So those do happen, and we've been targeted for some of those attacks as well, because they're an easy way for an attacker to get to extract 10s of thousands of dollars from these companies.

Grant: Yeah, I'm still waiting for your CFO to wire me that money. So hopefully, he'll get that into my account.

Alex: No he's well trained.

Grant: One thing you said earlier that I actually, I want to go back to like an architecture and infrastructure question for you internally, like you mentioned, we need to be up when you're down.

And so you have to take some additional layers of redundancy that most companies probably don't think about, even from the perspective of DNS, or like MultiCloud, so if AWS part is down, or some Google, load balancing networking thing is down, how do you approach that inside of PagerDuty?

Alex: Yeah, that's a great question, because this is something that we were focused on from our early days is how do we make sure that we architect a distributed resilient systems so that we can actually, one of the early design goals was to be able to survive a data center outage.

So we're using AWS and then over time, we added a second provider too so now our AWS and Azure. And we can survive.

I think like a full availability zone goes offline, which, in the early days of AWS, it did happen a couple times.

Nowadays, AWS has gotten much better, but that was one of our earlier design criteria let's make sure that this the system is active, active or parts of the system now that were service oriented and with lots of micro services, so part of the system is active, active part of the system is active with a backup and we can do a quick failover.

And I would say we think of the system in terms of what are the configuration parts of PagerDuty, like things that you would configure and then kind of change once in a while.

And what are the real time aspects of PagerDuty so things like us ingesting alerts and events from a variety of different tools, processing those making those into alerts and incidents.

And then once an incident is created, making sure that the right people know about it, whether they get paged, or they get notified in some other way.

So dispatching, SMS and phone call and push notifications and all of that. That's our real time portion of the system.

And we have different availability requirements based on what part of the system we're talking about.

So for those real time components, that's a tier one very business critical portion of the system.

So that needs to be basically, we target four nines of availability in that meaning that the number of requests have to be, 99.99% have to be processed successfully.

And really, our bar is even higher than that. We're like, we're targeting like five nines, but that's so hard to achieve.

That our kind of internal goal is four nines. But yeah, that's a really important aspect. And then over time, we also added the availability of the UI to the same level.

Like SP four nines available because a lot of our customers are relying on the UI and when, let's say notifications work, but the UI is down.

Grant: How do you handle it?

Alex: Yeah, they can't see, okay, who's on call? I forgot Bob's phone number. How do I look it up.

But PagerDuty is my directory for everyone's up to date contact information.

And if the app doesn't work, or the web UI doesn't work, then I can't look them up, I can't do anything.

So that's kind of a problem too. And our customers don't like that. So then that's a very high availability, it's also a tier one service.

And then we have other components of the service, maybe one of our newer products doesn't have so much usage or something that doesn't necessarily need to be real time available all the time.

So say a reporting module. Well, if it's down for like, 10 minutes it's not good, but it's not the end of the world.

You can look at the report a little bit later. Whereas not getting paged for a critical incident is kind of a major deal. So that need to work.

Grant: Okay, interesting.

Alex: But yeah, active, active in a lot of our parts of the system and for the notification components, so like sending out SMSs and making phone calls, and even push notifications and emails, we have lots of redundant providers because--

Grant: No longer just Twilio?

Alex: No, no, no. Even from early days, like Twilio, first version of Twilio, didn't send SMS, so we had to use more than Twilio anyways.

So they added SMS pretty quickly thereafter, but we're using kind of that earlier iteration.

So we have something like four or five different phone providers and like eight or nine different SMS providers and multiple email providers, multiple DNS providers.

And we can actually do things like in India, if this network like their version of Verizon doesn't work, then we can flip it over to a different provider.

Grant: Oh, wow.

Alex: So we're pretty sophisticated. We have a rules based engine that can do that.

And it can also detect if the SMS or the phone call didn't go through, then we can try a different provider and failover so a lot of work has gone into that for us.

Grant: Yeah, it sounds like that wasn't part of the initial version 10 years ago. Over time, it's become really important. And that's cool.

And I'm sure that these are the kind of gotcha questions that customers have asked over time and you just keep adding in deeper and deeper and more and more and people are like, "Okay, yeah, that answers all my questions."

Alex: Yup, I remember early, early days, like 2011, when it was just less than 10 of us. We had a problem with one of our providers. I think it was Twilio?

Grant: Yeah.

Alex: But I don't remember for sure. And what we ended up doing the volume wasn't that high in terms of how many customers we had and the pages that were going out all the time.

So we ended up pulling out the admin console and making the calls ourselves.

So we were using a whiteboard, writing down the phone number, like calling the person saying, "Hi, it's Sally from PagerDuty. "Well you actually have an incident," right?

Grant: That's amazing.

Alex: "Our phone provider doesn't work, "but we're handling this manually for you."

It was kind of cool and a lot of our customers early customers got a kick out of that.

Grant: That's amazing. And how long was it out for like hours, or?

Alex: It was maybe an hour or so.

Grant: Okay, that's great, I mean you figured it out right like hey, we've got their phone number I've got a phone.

Alex: Exactly.

Grant: Let's call them.

Alex: I learned that my phone can't call internationally without calling like Verizon and enabling that feature but hey.

Grant: It's racked up some phone bills too.

Alex: Yeah.

Grant: Cool and then you kind of mentioned some other products.

Is PagerDuty like a multi product you have multiple products, or is it kind of the core product with some add ons? How do you think about that?

Alex: Yeah, it's a good question. So we started with one single product and multiple tiers, like going back to the good, better, best, packaging and then over time, we added additional products.

Now, I think I would call them more of an add on like you need to have the base platform and then you can add additional products on top so the base platform handles the incident management and on call piece of it.

So everything from ingesting events, processing them, creating incidents, making sure that the on call folks get paged quickly and then we've added additional modules.

So one of the modules is around stakeholder communication and what we call our business communication modules.

So this is around, well, you have folks inside the business that are not on call, they're not hands on keyboard fixing problems, not engineers, but they need to know what's going on when there is an incident or if the service is being impacted.

So think anyone who's in management or maybe on the exec side, maybe anyone who's customer facing like sales or support or customer success, they're going to get potentially calls from customers, and they shouldn't learn from customers that they're having a service outage.

So it's about making sure that anyone who needs to know about an outage or an incident or a major incident that's impacting the business knows and that they know what the impact is. They know that the right people are working on it.

And they know there's a constant proactive way that they're getting updates, because otherwise what can happen is these folks can, like let's say the CTO, I'm a CTO.

So let's say the CTO learns about an incident happens to learn about an incident in a non proactive way, then they can end up jumping on the conference bridge and asking a whole bunch of questions where there's a whole bunch of engineers on that bridge trying to resolve the incident.

And they end up derailing the resolution effort, because they're asking all these questions. And they're, they're well intended, but they're not helping.

They're, actually distracting away from that resolution effort.

So we believe from a best practice standpoint, that there should be a line of communication that goes around the team that's actually responsible for resolving the problem.

And that should be handled through a process.

And there should be a separate process for business communications, and for letting the stakeholders know what's going on in a proactive way.

So that's one of the modules and then there's one around event intelligence. Sorry, AI ops solution.

So this is all around being able to predict incidents before they become these big outages based on past incident data. It's also around noise reduction.

So if you have all this alert, noise, event noise, what matters and what doesn't, and being able to kind of triage this and realize oh, this is just a transient thing that's not really that important.

Whereas this really matters and being able to prioritize that and do the right thing based on that.

Is that an all hands on deck like major incident that needs multiple teams? Or is this something that you can look at tomorrow?

You don't need to be paged in the middle of the night for.

Grant: And how did you decide to make these kind of separate products versus features? How do you differentiate?

Alex: Yeah, it's a good question.

I mean, part of it is we do a lot of customer research, and we do a lot of customer interviews to ask them, "Hey, do you want this? Do you need this? What problems would to solve for you?"

We've actually in the past also relied on a firm that does a lot of this pricing research as a consultancy.

This is what they do is they help you do this pricing analysis and help you understand what's the perceived value of a product, and how much you can charge for a product.

We're actually going through one of these repricing exercises right now where we might actually go read bundle some of these.

So, spoiler alert, we might end up taking some of these products. And maybe then baking them into the core platform. We're not sure yet. But that's definitely a possibility.

Grant: Okay, because today, it's like you kind of have these different tiers. And then you can add on these four different products A La Carte.

Alex: Yeah and it's also a matter of, what do most customers care about?

Because if they all kind of fall into the same bucket, or there's a lot of similarities, then you can just create a package for that.

But the A La Carte model has its advantages too, because if a customer doesn't value a thing, but they have to pay for it anyways, because they want like one feature, let's say it's a package that comes with 10 different features, the only want one feature, so then they're angry that they're paying for these nine other things.

And then they ask you, "Can you make a custom plan just for me, "so I can pay less 'cause I only want this one feature?" So you end up with those covers.

Grant: So true. And they're like, "I don't need that one feature. "Can I have a lesser price? "For that same thing minus that feature?"

And you're like, "Not really, "its kind of all part of the package." You put these things together. I know you don't necessarily but you're going to need it at some point.

Alex: Yes,

Grant: It's funny. Pricing is hard, pricing is like it's never--

You're never going to nail it 100% and, you try to capture some value that you're providing, without like, feeling extractive or like overly, no one wants to be Oracle, in the world where everyone feels like Oracle is just like taking their pound of flesh every time.

You want a feeling of reciprocation, or like you're giving somebody something that they love and appreciate. And they're reciprocating by paying you, right?

Alex: Yeah.

I've never been a fan of extracting the maximum amount of value out of your customers because then you're vulnerable to competition and at some point, the music will stop and if they don't see you as a partner, and they don't like working with you, and they see you as kind of the enemy, then they will switch off of your technology the as soon as possible stage that they can.

Grant: Yeah, 100% agree. It's just better. I mean, all negotiations leave a little bit on the table.

Alex: Yeah.

Grant: The other side should never feel like they got totally hosed. And you should never feel like you totally won.

Because then, like, everyone feels like yeah, I gave a little bit but I also got a little bit.

Alex: Yeah, exactly. Because relationships matter and you don't want your customers to be grudgingly give you the money and then hate you.

And like switch off. Again, they'll switch off as soon as they can.

Grant: Yeah, it's like long term, you're creating a bad dynamic with your customer.

But even short term, it's just like, it's not worth the extra stress and the extra pain and, you want to see a customer and they should be excited to see you right?

They shouldn't know walk in the direction or say, bad things behind your back. So yeah, I totally agree.

But it's, again, it's hard 'cause you can get a lot of value out of a product with very little usage.

And you want to capture some amount of that value as a provider because that's why you run a business you have to capture some of the value.

So you've obviously grown and learned so much from your time at Amazon as an engineer to becoming a founder, CEO, CTO, taking your company public this whole like trajectory.

When you kind of look back, even from going from someone who probably just thought about this as an SMB mid market thing to like true enterprise, what's maybe a key thing that you've learned?

Or if there's some advice you want to impart on folks listening that would kind of wrap some of that learning up.

Alex: So one of the things that I talk about and go to a lot is, is the importance of culture and being--

Culture is something that forms whether you like it or not, s o my recommendation is to be very intentional about it and start thinking about it and investing in it from an early stage.

Like when it's just the say the two or three or four founders, you probably don't need to start spending a lot of time on it then.

But once you start hiring people, then it really starts to matter.

And it's good to have a set of company values that you believe in that are not just things that are written up and posted on the wall, but things that get operationalized that matter when you're making decisions that matter when you're hiring people that matter when you're, say doing like, yearly performance assessments or giving raises or looking at people's roles, or all those things as are they actually embodying the cultural values?

And I think it's okay to have a set of values that are true today that are, you're following and you're living every day and then maybe have like one or two that are more aspirational, like you want to embody but you're not quite there yet.

And then you kind of put, start putting a plan in place to do that and to get there.

Like let's say you didn't invest a lot in customer support early on, but you put a value around taking care of the customer, putting customers first, customer centricity, something along those lines, then yes, something you could start building into the company and investing in over time.

It doesn't have to be true and everything doesn't have to be true today.

Grant: I love that that's a great point. It's like it doesn't have to be a snapshot of where you are.

It can be a combination of what you are, what you're doing and where you want to go.

Alex: Yes, absolutely. That's one I would say another one is around goal setting.

So early on, we used OKRs objectives and key results. Now we're using a different framework called V2MOM, which is a Salesforce framework, which is very similar to OKRs.

Except it also encompasses like the V2s vision and values. So it also talks about that.

And it doesn't really matter which framework you use, as long as you use something.

And it's the simple fact of, what are you going to do in the next time period, let's say it's a month or a quarter.

The smaller you are, the smaller your time period should be you shouldn't plan for a quarter out to maybe do a monthly or even bi weekly, and look ahead of what you're going to do, write it down.

And then at the end of the period, compare what you actually did against what you said you were going to do and see how that stacks up.

It's a great way to build accountability into the culture from an early stage and make sure that everyone's aligned around goals and what's really important.

And then, like resources are limited. So it also helps you focus and prioritize on what's really important and helps you inspire the folks are around rallying towards the common goal.

So if, let's say we start with-- I'll give you an example from PagerDuty.

We started with on call management and alerting, and we want to make a big push into event management and be able to process all these data sets and add machine learning and smarts and intelligence into the equation then, maybe 2013 was the year of event intelligence.

And we put a big focus there in our goals around launching a product and getting say 20 early customers using it and five testimonials.

I'm just making stuff up here. But yeah, like in order to kind of prove out he early success of that product.

Grant: Yeah.

Alex: And it's important to do that and to also, it's part of holding yourself accountable to your investors and showing them here's what we're planning to do for the next quarter.

And even thinking out a year ahead in terms of a more strategic longer term trajectory, like here's what we want to do this year 'cause you certainly have to set financial goals, especially if you have investors and share those with your with your board and your investors.

And you have to break that down from okay, what are we actually doing?

What are the initiatives that are going to lead to us being able to hit those numbers that we set with the board.

So that's another thing that's important, I would say, just on a more personal note, being able to invest in yourself as well into your own self learning.

I think, I wish I had done more. I've done some in my past and my last 10, 11 years of PagerDuty at times, I've had a executive coach.

I've built a network of folks around me who have set of advisors.

I never got to the point where I built any mentorship, mentor mentee type of relationships, but I wish I had honestly.

But being able to invest in that and invest in self learning, because remembering that I started as an engineer I had no management experience, not a lot of hiring experience.

Not a lot of like hiring executives, especially hiring execs who you don't know their domain as well as they do.

So how do you know that they're good at their job? Those kinds of things are tricky.

Grant: You didn't hire any execs Amazon?

Alex: No, as a software engineer, no.

Yeah, that's actually one of the pieces of advice from one of our investor really stuck with me, which is to do a calibration process for let's say, you're hiring a VP of sales or head of sales.

So the idea is to do a process where you they introduce you to a whole bunch of VPS of sales, ones that have been successful that are known to be good.

And then you talk to them, you sit down and talk to them and ask them like, "Hey, how do I hire one of you? What should I ask? What interview questions should I ask ? Are there different flavors of VPS of sales?"

In this case, yes, there's probably ones that are more operationally focused, more numbers driven more sales ops like that's their strength versus there are folks who are more inspirational who can rally the team, especially at the end of the quarter to get everyone excited and motivated.

And there's probably like multiple archetypes in each type of role. So it's learning like what makes a good VP of sales in this case.

And how do I evaluate that person. And then actually maybe even leveraging some of their advice and expertise, so your investors advice and expertise in the interview process.

So having them do interview some of these folks as well, just to double check, and then reference checks.

So that's another piece of advice that I've gotten. That's been super useful, 'cause, I've never run a sales team myself.

I've never headed a marketing department before. I've never been a CFO before. So how do I hire all these people?

Well, you just do a learning process at the beginning as part of that calibration, and then figure it out, and then learn and then actually execute on it.

Grant: That's all great. And I would actually I'd take one issue, I'd say that you have established a bunch of mentors throughout your time.

I think all these people, you're talking about these folks that have been advisors and investors and they've helped sort of guide you along the way.

Alex: Sure.

Grant: I mean, that what it is, doesn't have to be some one formal person these are these are all people that have helped you kind of grow and expose you to new thing in sales.

Alex: Totally, I just never had anyone that I called my mentor.

Grant: Yeah.

Alex: I guess maybe I'm thinking of that relationship as a more formal relationship where I pay them or give them a little bit of stock.

And I've had lots of advisors, but nobody's been like a formal mentor.

Grant: Yeah.

Alex: With the title.

Grant: Right but all these people. I mean, they've all been in that role, which is great. It's kind of like a community of people.

And I'm sure, as much as you've referenced them, and you've kind of called them out for what they've helped you accomplish.

Sounds like you're super appreciative of what they've done for you, too.

Alex: Oh, yeah, absolutely. And I've always taken the approach like, okay, I'm a first time entrepreneur, I need to learn something new. I need to do something new.

So let me start by learning and surveying and seeing how other companies have done it because I'm sure I'm not the first to figure this problem out.

So let's start there. Let's learn and then let's execute after we know what to do. Let's not just charge ahead blindly.

Grant: Well, thank you so much for sharing all of this knowledge with, everyone that's listening.

Hopefully they'll be able to do that same thing with their companies.

Alex: Yeah, absolutely, thanks for having me on.