Ep. #27, Ready For Observability with Corey Quinn of The Duckbill Group

Corey Quinn: Yeah, I tend to view, for what I'm doing, observability as being analogous to hipster monitoring.

Because fundamentally what I'm running isn't scaled out enough to really be an observability challenge.

I need tail-F on the logs on a continuing basis. Maybe if we're being fancy with the grep thrown in but baby seals get more hits than most of the stuff that I'm working on.

So it's not like I have to extract a whole lot of signal from noise.

Charity Majors: So this is an interesting question.

I feel like now that there is a bandwagon everybody's jumping on the bandwagon, every one needs observability. But they don't actually.

Sometimes monitoring is what you need. Sometimes what you need is tail-F with a grep thrown in, right?

I think that in order for us to be, you know consistent about the fact that there is a technical definition for observability, it is distinct from these other things.

It's important for us to be forthcoming about when it is actually needed.

Corey: Well, I think that like the old stories of systems administrators became DevOps people slash SRE's which I'm sure no one here is going to have a problem with that characterization.

Charity: No opinions whatsoever.

Corey: My argument is that it's fundamentally the same role, the tools that you use and how you approached it change radically, but to some extent, the core definition of keeping the systems in production up and running long enough to make money is kind of the distillation of the job.

The rest of it is just marketing.

Charity: The paycheck changed too. Don't forget the paycheck.

Corey: Bingo. There's a reason I don't call myself a systems administrator when I'm wearing that particular hat because should DevOps be in a job title, well, that's a great theoretical discussion.

Should my paycheck be 30% larger? Well, that's a bit more nuanced.

Charity: So, I actually think that the step function here was adding engineer as an appellation. In my mind, there is something actually different.

An engineer builds something, right? An administrator runs something.

I feel like this is the stream that split that never should have split into dev and ops and is now like rejoining.

But it means that, you know, Ops people, yes, need to learn to write code. Devs, yes need to run their system.

They should all be engineers. And so I do think that there was a bit of a step function there and I'm being optimistic here and trying to give the benefit of the doubt but like that's what really deserves the pay difference.

Corey: Yes and no, because on some level it feels like when you go too far down that path you wind up with early optimization out the wazoo.

There's a great picture on the internet that goes around from time to time of a flatbed trailer with a single tiny box on it.

And it's always captioned, "Yay, my blog is now running on Kubernetes.

Liz Fong-Jones: That's great.

Corey: It's a common problem though where people have find this great amazing tool that does incredible things.

And what do you have to do now? We'll go find a problem that I can shoehorn this into as an improper solution.

Liz: That sounds also a lot like the idea of people just saying we need Site Reliability Engineers because Google uses Site Reliability Engineers.

It's like, no, Google is a very large and growing company and has lots of services. And you may not necessarily be in that position.

Corey: Absolutely. I mean, be like Google is a fairly fraught statement. They have a different scale than most of us.

They have a different reactions to some things.

I mean, most companies generally don't wind up paying tens or hundreds of millions of dollars in severance packages to people who are leaving under a cloud of sexual harassment.

But you know, that's just my personal take on it.

Charity: They're special.

Corey: Yeah.

Charity: Google's special.

Liz: So when do you need observability then, Corey if your kind of seal sized application doesn't need observability, when do you need observability?

Corey: For me it's always come down to you need it at a point where the signal gets lost in the noise.

Where you have hundreds of transactions per second in many cases, depending on what you're seeing.

And one user is having a bad time, great. Trace through what they actually did. In many cases it's, "Oh, that's a payment flaw."

We should probably care about errors that thing throws versus the ad rendering service.

Charity: I think you definitely need observability then but I would actually back up a little bit and say, you need observability if you're writing and shipping code regularly.

Now I'm guessing your steel doesn't get a lot of constant updates made to it.

And so it's pretty easy to understand and to track down what changed when you did something wrong.

I think the observability is the magnifying glass that you hold up.

It's this code itself explaining out to you. Here's what I'm doing. Here's what it looks like. And letting you follow along.

And anybody who's making a lot of changes to their application means observability.

Now, black box software like if you're just stringing together components that you cannot change, are not going to change.

You know, you're going to do a package upgrade once a year.

You don't have observability for that code and you probably don't need it.

You need it for the code that is your differences as a business.

Corey: Yes and no. Because remember that when it comes to writing code, I am god awful at it which means that most of my development flaws don't resemble anything that you would call a good practice.

I wind up deploying again and again and testing it and where it lives in. Great.

I keep iterating through and wow. I shipped code to production 200 times yesterday.

Yes I did. That doesn't mean it's a good thing.

Charity: So maybe what we're coming to realize is you do need observability. You just don't have it.

Corey: Yes and no.

Charity: Is that your answer to everything to literally everything

Corey: It really is... The answer to everything on a nuance level it depends.

Liz: Speaking of which, now is a time for you to introduce yourself.

Corey: Sounds good. I'm a Chief Cloud Economist at the Duckbill Group, Corey Quinn.

And introductions are always hard because it really comes down to what do people associate me with?

Liz: So your answer to this question is yes and no as well?

Charity: Yes.

Corey: Absolutely. I should post on twitter constantly. I read a sarcastic newsletter called Last Week in AWS.

I host two podcasts. Screaming in the Cloud and the AWS Morning Brief.

I am the world's greatest cloud influencer as decreed by some random company at the beginning of the year.

And I guess most near and dear to my heart, because you know, it affords to keep me in ridiculous ponds is my work with the Duckbill Group the company I founded that fixes the horrifying AWS bill.

This is an ongoing challenge. I'm different things to different people.

And when someone says, "Who are you and what do you do?" It's very hard to get a line around an answer that isn't, "Well, just sit right back and I'll tell a tale."

And suddenly you're doing an entire season of Gilligan's Island.

Charity: To be clear, we said, "What is your name?"

Corey: Oh, I see. Corey Quinn.

Charity: Okay.

Liz: Okay, that one has a straight forward answer that isn't yes and no.

Corey: Well, I do have a maiden name. So it's more complicated than you might think.

Charity: Absolutely. As it should be. So recently I hear that you were promoted to Chief Cloud Economist.

What does it take for a young chap or a young lass who wants to grow up to be like you?

Corey: A series of poor, bad life decisions that continues to escalate.

The reason that I added the Appalachian Chief to my title was because I was told to.

We now have multiple Clouds Economists who work here at the Duckbill Group.

And I'm trying as best I can to shield them from the fallout of what did that Jack wagon say again?

No, no, that was the other Jack wagon. The one who's stupid grinning face is on the website.

Charity: Well, you want them to have career trajectories too?

Corey: Exactly, and have a career trajectory that ideally for most folks, is it into the other side of the mountain.

Charity: That would be nice. Does this mean you're doing more management these days?

Corey: It comes and it goes. It really depends on any particular month or quarter or what's going on.

I mean, going back to the whole story of all the different things I do a lot of my function at this company is being incredibly self-promotional in public and that's great.

It's valuable. It definitely is a rising tide that lifts all ships or drowns everyone else.

But it also means that you're not able to be promotional as much of the people who you work with, which from my beliefs has always been the primary role of management.

Liz: Yeah, that's kind of challenging, right?

And I think that, that's part of why it was super interesting the role that I came into at Honeycomb as the principal developer advocate because now there's more than one developer advocate.

We have Shelby on our team and you know, indeed it's this challenge of being self promotional, being a individual contributor but then also kind of a coach, a technical lead.

These are definitely challenges to navigate.

Corey: It gets worse than that because--

If you're hiring effectively and well, and you start off as a one person shop as I did, then every person you hire for the thing that they're doing they are definitionally going to be better at that thing than you are.

Liz: Mmhmm - Yeah.

Corey: Which leads to a story very quickly of "Wow, I hired someone" and wait now just tell them how to do the thing?

That doesn't make any sense at all. So if you start down that path suddenly you're effectively micromanaging them badly.

Whereas if you're too hands off and not communicating things like vision strategy, why we're doing the things we do, you look at what they're doing and "Oh no, no that's going to get us sued "at least four times this week."

So there's a happy medium in there somewhere. And striking that balance is always a challenge. Anyone who says that they solve for management is trying to sell you something.

Charity: Well that's like saying that you've solved for people.

Corey: Well, I don't know.

I've met a lot of engineers who would claim that they have but it turns out that you can't install an API into people legally.

Liz: Unless you're an Elon Musk.

Charity: That's a great punchline.

Corey: Yeah. I love that. Bill Gates is talking about vaccinating people.

No, he's going to wind up putting microchips into people. Great.

Elon Musk wants to put microchips into people. How do I pre order one of those?

Can we get a little internal consistency, please?

Liz: Just think about what their observability needs must be.

Corey: Yeah. Project Panopticon. That's going to go super well.

Charity: Well, we're on our way. Yeah, the question of management when you're--

So for me, I have never thought of what I do as being self promotional.

And so it's been weird, like having to--

I think that we've been navigating sort of similar waters there where, you know now there's a team and am I taking up too much space and too much airspace.

And for me, part of the struggle has just been starting to become aware of the airspace that I am taking up so that I can consciously take a step back and let other people forward.

Corey: One area that might resonate a bit more from that perspective is I believe you and I suffer from one of the same weaknesses, which is honesty.

Now, some people may say that honesty isn't a weakness to which my response is I don't give a fuck what you think.

Charity: Exactly.

Corey: See, there we go.

But that means that, there's a very real chance that my aggressive shit posting on the internet has the potential to damage the career trajectory or income potential people will put their trust in me.

Charity: To get people you love in trouble.

Corey: It's a hard balance to strike.

Charity: Yeah, it really is.

Liz: Yeah, I think that it depends upon the environment.

One of my core values is radical candor, right? That feedback is a gift.

So, I think that that's something that we can do internally but indeed like when you cross those external boundaries other people may not perceive feedback as a gift that we work with.

Corey: How do you keep radical candor from turning into effectively an excuse to just dump on people. I've seen that manifest a couple of times in other places.

Charity: Well, part of it is like you know, have people who have some sensitivity and part of it is exposing them to the consequences of their words.

You know, it's really easy to shit post to the internet because you don't see the pain faces of the people who you're hurting.

When you're sitting in the same room with them, or when you're when you have to talk straight to their faces.

This is why managers have a really hard time giving their teams the feedback that they might desperately need in order to grow, but they're right there in your face.

And so you could see how much it hurts them and it becomes really hard. And this has been something that has been challenging for me, in that I show my feelings all over my face.

Like I have no poker face. I don't even know what poker face means. And Liz is nodding right now emphatically.

And that means that I have had to start to realize that it can be really hard to give me feedback that even when I want it, even when I'm asking for it, even when I'm telling you that I want it, you can see how much it hurts me and that makes it hard for you.

Corey: It's a challenge to get it right because there's a strong element of empathy because without feedback people don't grow.

But I personally have a tendency to look at something that is 95% awesome. Great. We accept that.

We know that. It may not even be spoken of. Let's talk about the 5% that we can improve into each year.

Charity: Let's talk about the edge cases here that--

Corey: Right, so you sound like you're never satisfied and people hate working with you if you left your own devices on that.

Charity: So I try to compensate for that by just like--

One of my favorite things is repeating compliments for people.

Like my co founder, Christine is wonderful at so many things but she's not good at telling people how amazing they are.

So every time Christine compliments someone to me, I rat her out.

I tell them, "You'll never believe what Christine was saying about you. She was just saying how amazing you are and how well you did at this thing."

And I find that really gratifying, just like telling people all the amazing things that other people are saying about them.

Corey: Yeah, it's hard to do that.

Liz: Yeah, I think part about radical candor is that you can't selectively practice it only when they're bad things.

I think you have to practice it when there are good things too.

Corey: Right, and the idea of, "Oh, just do the feedback sandwich."

The negative in the middle between two positive things.

Charity: No, it doesn't work.

Corey: And then people phone it in too it's like, "While, you dress very nicely. "Your work is crap, and you're very emotional."

Charity: Exactly. Yeah. Just shortening that feedback loop and making it so that you're doing it quickly after it happens so that it has maximum impact.

So that people emotionally connected to the thing that they just did, whether it was good or bad, it's just like trading puppies really.

Liz: Hey, wait a second. That sounds a lot like what we practiced in terms of software too.

And that sounds a lot like reveality method of getting the feedback rapidly around.

Charity: Yeah, that's true. Has to be near real time.

Liz: Yeah, because if it's not real time then it's very delayed and then you don't know.

Corey: Oh yeah.

Charity: Very, very hard to correlate.

Liz: And I think that, that's true a lot in terms of your business too, Corey, right?

Like if people don't know what the consequences of spinning up, that VM is going to be, then they're just going to spin it up and their CFO gets mad at them like two months later, right?

Corey: Yeah, there's even an observability story here in that, when you don't measure things and are able to do a baseline comparison till we made a change, how do we measure whether it was a good change or durable change or something in between?

It's a tricky thing to figure out. Lord knows I've gotten that one wrong a lot.

Charity: Yeah, So I think that if we're making this analogy, the way that we gather our telemetry from human beings is by talking to them and by asking them how they're doing.

We may not have API's but we do have the ability to extract information and just get used to seeing patterns and this is interesting because we're hiring for the first time a manager from outside, like we've always made bad--

Not promoted, but we need to do it for work. It's like, we've always transferred managers from within. And so we know that they're very Honeycomby.

It was really funny. We've had executives start working here and they're like, "You guys talk about your feelings a lot." And we're just like, "Yeah."

And when they're engineering interviews like we select very highly for communication.

Because we figured that everything else can be taught, right?

You need some basic skill level so that you are at the place where you can start learning the thing, right?

We can't just take people straight out of high school.

Who've never talked to computers but people who can communicate well about what they're doing.

And it's funny because we started to notice that there are companies where people who have worked there are coming to interview here and they almost categorically don't succeed.

Because those companies don't have a culture of talking their way through their code, or explaining what they're doing out loud.

And whereas their code might be fine. That's not actually what we interview for.

We interview you for your ability to like talk me through what you're doing.

Why did you do it? What else did you think about? What did you consider? What are the trade-offs? That's what we interview for.

And it is not the same as just being able to write the code.

Liz: Corey, so you said that there are more Cloud Economists at the Duckbill Group now.

How did you wind up interviewing them? I imagine it has kind of a lot of those same considerations of like, do you know how to talk to people?

Corey: Absolutely. People with a-- One of the things we required was a background in consulting, at least for the first few hires.

Charity: Oh interesting.

Corey:

It's easier to teach people who've already been in that type of environment and understand the context of customer satisfaction than people who have not had that experience.

We also looked at this from a lens of what is our functional role as a company and realistically it's marriage counseling for engineering and finance more often than it isn't.

So having people who are steeped in the engineering background side who have done SRE style work by the common SRE definition, I know, I know, who have been exposed to the technology side and have a deep understanding of AWS.

It's easier to teach those folks the nuances of finance than it is to go the other way around because, "great you have a background in financial planning. Cool. I'm going to teach you this thing called Terraform now."

And that it does not go nearly as well.

Although if you get a few people who are enterprising involved you can almost certainly have someone managing all of their AWS estate via Microsoft Excel.

Liz: That's a controversial statement. People can manage everything with Microsoft Excel.

Corey: It is the world's most common and broadly deployed IDE and User Interface.

Charity: And database.

Corey: Oh, absolutely.

Liz: So I guess that gets us to the question of, when do you build your own custom AWS measurement thing versus just using Excel, right?

Like when do you build versus buy?

Corey: Great question. We do a lot of stuff with Tableau which is sort of a weird half measure between this.

It comes down to taking the data that they give us from various accounts, cost and usage reports and slicing and dicing those in ways that are effective and easy to work with the distill again, signal from noise.

The idea of building our own analysis tools on top of that, well, yes and no.

We have some tools that we've built that wind up solving for particular analytics work looking at every DynamoDB table and figuring out does this one make sense to have an on-demand mode or provision capacity mode, for example.

That winds up being a very specific use case. And there is nothing off the shelf that does that. And not for nothing.

It didn't take that long to have something like that built out effectively and well.

Whereas, "Oh we're going to build a whole dashboard "to manage all of AWS."

Great, good luck, have fun. People have tried this end up more or less kind of out of business.

That was RightScale's whole thing back in the day when DC2 console was garbage.

And now that it's less garbage well, people aren't willing to necessarily pay a third party to paper over that anymore.

Charity: You know, when I was managing interview at Cast last time I managed it all with a bunch of Python scripts just ran them once a month and paid the bill for the difference. Yeah.

Liz: Yeah. So it sounds like basically there's this idea that good enough is usually fine and that many different tools are better than trying to build one tool to solve it all.

Charity: Well, I mean, this goes back to the fact that like everything that you do that is not your core business takes away from your core business.

And something I've been realizing just recently is that it makes sense that what we have is a bunch of monitoring tools because monitoring tools are the right tools for infrastructure, for building and maintaining your infrastructure capacity planning all this stuff.

And up until very recently the majority of engineers at every single tech company work on infrastructure, not on the stuff that makes them, them, right?

Their core differentiators. The code that like is their reason for existing.

Like think of all of the people that we had, they have like racking servers they're running databases, configuring them.

And then you know, people who are writing libraries, people who are you know, like the number of engineers, and this is insane because developer cycles are the scarcest resource in our world.

And this is where I feel like now that some tech companies have reached the more than 50% of their engineers are actually moving the business forward.

It becomes glaringly obvious that monitory tools are a terrible fit for observability problems.

But you need observability for your business differentiators. You do need monitoring for your infrastructure.

And I think that as we see those ratios start to trade place, I think that becomes more and more apparent.

Corey: I don't envy you, in that, when I explain to customers what I do in terms of the consulting business, media business is a separate argument there, but it's very straightforward.

I help fix the horrifying AWS bill. I let you spend less money and get similar or better outcomes through what you do spend but that's it.

It fits in a tweet and we're done. You on the other hand have to fundamentally define terms like observability to folks who haven't heard of it.

To those of us who think it's all just monitoring.

Charity: High cardinality.

Liz: And I'm not necessarily sure that's true. The outcome is that engineers are more productive. How we do it is observability.

Corey: Right, but it requires almost a lecture to some extent.

Whereas now I've spent 45 minutes explaining how this works why it works and the rest of the end of it, it ends with a sales pitch and it's hard to get there.

Charity: Which is why we've spent the last four and a half years doing it.

And now we don't have to explain it to most people.

We just have to explain how most people who are using our language are not actually solving the same problems which is a different problem.

And one that we're starting to Segway into it just like "okay, now that every motherfucker out there "is like we need observability."

Well, okay. But like let's show how they don't but not by focusing on them, but focusing on the things that we help you do that are important that make a difference, that their tools just can't do.

Liz: I look forward to the day that all the companies who are kind of selling software around Right Sizing AWS instances to advertise themselves as Automatic Cloud Economists. It's going to be hilarious.

Corey: Oh I'm waiting for it. If I thought the answer to this was just a SaaS product that was not going to solve problems for this.

Great. I would have built one of those. In fact in the early days, I thought that's where I was headed.

And then I started talking to customers and Oh yeah, there is no way to get that insight.

Charity: There isn't because these systems are almost as complex as human beings and the way that you can't solve people you can't solve systems.

Which is why it bugs me the way that people have been telling everyone that you don't have to instrument your code.

You can just install our library and never even think about it.

I'm sorry, your business case is only understood by you only you know what you were trying to do, what trade-offs are involved. And they're like, we can make it easier.

We can provide helpers, but the person who has that in their head is just like, how many years have we wasted trying to auto comment code.

You can't do it. It is exactly the same problem. Instrumentation is commenting your code is instrumentation.

They are the same fucking thing. Just one involves production. One doesn't.

And these systems that we've built to solve these business problems are so customed, so unique, you know, or they shouldn't exist like by and large, the market weeds them out quickly.

If there isn't enough differentiator there to exist and they don't exist.

But if there is, then it's a human scale problem. It's not a computer scale problem.

Corey: Yeah. I wish I had a good answer as far as how to get people up to a certain level of understanding.

On some level, this is always going to be a problem for any realistic product or service. It is targeted niche.

Charity: But the answer is, I don't think this is niche at all.

I think everyone who writes code needs to care about this honestly, because they're wasting a half of their life at work.

They're wasting literally after life at work by not being able to orient themselves, figure out what the fuck they're doing, working on the wrong thing.

You would be surprised. Like everyone who writes code should care about this, but you're right, they don't need to get down into the nitty gritty.

Liz: We want them to have to worry about all of that, right? People need to understand how a column for database works.

Charity: They shouldn't. Our customers are the ones who can talk about this on our behalf.

They can say, "Oh my God, this was life changing." You know, that's it.

Corey: Oh, yeah. I'm talking in terms of what customers are experiencing and how the pain works.

It's always been effective.

One of my problems has been marketing to developers has always been that developers are generally a little on the, "We're going to build side things" because that is what they do.

I'll just whip up some code.

Charity: I have the hammer, let me bang something.

Corey: Exactly. And they're often highly skeptical of what they perceive to be sales pitches which I get having sat through a number of terrible ones.

And at the end of it you finally power your way through that force-field and that reluctance, great.

They're signing a 30 caps out somewhere around 50 bucks.

And it's a challenging sale to someone who doesn't have signing authority, which is why enterprise Salesforce is very often just tend to route around developers.

For what you do, I don't think that works.

Charity: No. It doesn't. In fact, we spent a couple of years trying that and I spent those years pulling the other direction.

That's why I'm not CEO any longer.

We're on the same page now because COVID happened and now nobody's signing these big checks, right?

And so suddenly everybody's on the same page for the first time in our history.

Everyone's on the same page. So I'm excited.

Corey: It's always challenging. And there's never going to be a global answer for all of this stuff.

There's always going to be nuances. And as soon as it's clear that there's money in a particular niche, brace for imitators.

Folks who have been in this space since Jesus was in diapers, now rebranding the same warmed over stuff with the new technology.

Charity: Oh my God.

I started getting like pitch decks forwarded to me from random VCs or people who are just like, "Oh my God, these people are just in here saying we're pitching Honeycomb for Kubernetes or honeycomb for blah.

And like, these guys could literally have been our attackers, so creepy.

Liz: Annotation is the best form of flattery for sure.

I wanted to kind of jump back onto the topic of, you know the developer who has less than $50 of spending authority.

Because I think that's kind of dynamic has played out in any kind of interesting way in our company, in terms of, I had a bunch of stuff that I was like super excited about doing with our AWS infrastructure and Charity was like, "Whoa, Liz slow down."

Like, you know, like prove to me that it has ROI prove to me that it has value.

And I think that that's kind of a interesting mirror of some of the challenges we face getting all in into companies.

Corey: There's no great way to go about shoehorning this stuff in across the board.

One of the challenges I have with all of this stuff regardless of whether if I'm in your stuff, my stuff, et cetera.

You've been a referenced customer of ours and we love working with you folks.

But the challenge has always been, it's a sophisticated sale to sophisticated buyers across the board.

Go to the biggest changes in my mind every year is when I do the Last Week in AWS charity t-shirt fundraiser.

Where we raised a bunch of money for St. Jude's and put something funny on a shirt and effectively sell that to folks because it turns out almost everyone has signing authority to make a $35 donation to a cause that virtually no one is on the other side of, kids with cancer.

That plays out to a very different dynamic because if anyone can reach into their pocket and pull out a credit card and spend money, great, I go to sleep and I look in the morning and I have a giant sea of notifications in my email where, "Oh, that's each one of them "billing a transaction report."

And I'm looking at this going, huh? In another year or two with the volume I'm seeing on this is my audience continues to grow.

I'm going to need the observability solution for the $20 T-shirt then.

So, what do you use for observability solution of a search in Gmail?

Charity: Yeah. There you go.

Corey: I just want to help Nagios up to my entire environment, but email me whenever anything goes wrong.

Charity: You know, I feel like Nagios is just about become retro.

Like I almost think people are just going to start spinning it up ironically, because they miss getting those pages in the middle of the night.

Corey: Oh, yeah. I want to get a port over to the Xbox because if you think about it, Nagios was the original Call of Duty.

Charity: Terrible. That was terrible. I'm sorry, we're kicking you off for that joke.

Corey: Excellent. Excellent.

Liz: So Charity what was your reaction when I was like, "Hey charity, I want to port our entire "production infrastructure to an entirely "new processor architecture?"

Charity: I think I said, as diplomatically as I could Is this going to move the needle for us the next six months?

And is there other stuff that you can be doing with your time that would?

Because you know, Liz is an info nerd and she looks around our infrastructure which is hacked together with duct tape and sand you know, and as you do, because it's the right thing to do, right?

It's working good enough. And she looks at it coming from Google and she's just like, "Oh my God. "Oh my God. "Oh my God."

And I'm like, "I get it." But like if there's one thing that start up kids are good at doing, it's not doing anything too much.

You know, it's just like looking at, Okay, our time horizon has risen and fallen.

There have been times that our time rises, we're building for the next two weeks.

If it's not going to pay off in the next two weeks we have no business spending our time on it.

And right now, you know our time rise, it was like a year. But like, for most of last year, it's been like a few months.

If it's not going to pay off the next few months it was not going to make our runway demonstrably longer.

Someone like Liz especially, has other things that she could be doing with her time.

Liz: Yeah. So I think that it's kind of interesting to see how does AWS sell graphic time two versus kind of how do we sell observability, right?

How do we persuade people that the investment of time and energy is worth it and going to pay off on the right time horizon for people.

Charity: And furthermore, I feel like anytime you're asking someone to do something new or learn something new or switch their tools or whatever, what you're offering them has to be order of magnitude better than what you're replacing.

Not 50% better, not twice as good or it's just not worth it.

The cognitive overload, that having to educate everyone, having to replace everything, do the migration, it's not worth it.

Unless it's an order of magnitude better. And I honestly think that when it comes to shipping software better, I think this plate has chosen the DORA metrics, year over year.

L ike in the last couple of years you just started to see the elite category to start like, go for liftoff almost.

And I think that it hasn't actually been an order of magnitude better until just the last couple of years with observability, with feature flags. But now it is.

Now it's an order of magnitude better if you're shipping software this way.

And so I feel like we could really bridge it badly and honestly.

Corey: The way I see it is there are two real things you can sell a company.

One, is what you're selling which is you're improving feature velocity. You're reducing the time it takes them to get features to market, to sell things.

Whereas the other side of it is what I'm doing which is cost reduction on the backside.

And that is never going to be as exciting as bringing a new feature to market. So the inverse of personal psychology around money.

Charity: I think there are many CFOs who would very much disagree with you.

Corey: Yes or no, but there's a reason technically speaking CFOs aren't the folks dictating corporate strategy.

Liz: But the CFOs are not in part of the engineering management, right? Like that's the challenge.

Corey: Yeah, because if I ask you as an individual to either, you have a choice, you can make another thousand dollars or you can cut your expenses by a thousand dollars.

Most people are going to cut their expenses because it's "Oh, I can cut back on Netflix "and stop eating out as much "and do a few other things."

Whereas getting more money needs that I negotiate my boss for a raise and I need to go ahead and dust off my resume and get a new job.

Companies are the exact opposite.

There's a theoretic form or bounds of what I can say if a company of 100% of their code bill, and the right feature to the right market at the right time can bring in multiples. That second option of improving feature delivery is far more valuable strategically.

At some point, you're looking at this bill and thinking, "Wow, that is the GDP of a small country."

We should do something about that. But it's never the number one priority for most companies.

Charity: That kind of like, for VC backed companies, at least I think that like companies that are tethered to reality with the firmer strings, right?

Liz: Yeah, yeah exactly. Because like, if you think about development as a cost center, then you think about the maximum savings being a hundred percent of your dev budget rather than it being a moneymaker.

Because we kind of haven't necessarily thought about this in terms of our client base.

But actually I would say like in every organization that we successfully come into, those organizations are all places where they view technology departments as an investment rather than as a cost center.

Charity: Source of creativity.

Corey: Yeah, and that's been a problem for a long time.

It's why generally speaking I tend to have much more success with companies where the AWS bill is empowering the core function of what they do and not back office IT.

It's for whatever reason, the stakeholders shift radically the personas are radically different.

And we have had successes on both sides of that particular divide, but the company is doing SaaS style stuff, is really our core bread and butter.

As far as we speak the language, we understand the constraints.

Charity: Yeah, that sort of makes sense to me because with IT you've already made your peace with this as a cost center.

It's just money I'm flushing down the toilet. Is it 60,000? Is it 80,000?

You know, you're already flushing it.

But when you see it as an investment that you see smaller tweaks relatively speaking as being worth it, because if you're investing in this place, that means you can't invest it anywhere else.

And so it's worth periodically rebalancing.

Liz: And also as you business expands it's going to become even more of a money hole as opposed just being a static sized money hole.

Corey: There is an ongoing series of escalating challenges as far as understanding corporate taxonomy.

Something I found is the bigger a company gets and the organizational distance increases between the person who cares about the bill the buyer, the person who has to implement stuff.

At some point, it becomes this incredibly convoluted sales process.

And fortunately, I have excellent people now who are very adept navigating that in ways I never was.

If I can't close a deal in 20 minutes on the phone, then it's not worth chasing.

Turns out that's going to cause some problems in the enterprise sales market.

Charity: An ongoing sequence of escalating challenges is definitely my band name.

Corey: Yes, that's also kind of ideally everyone's career track.

Liz: Kind of makes me think a little bit about like you know, when I think about how to help AWS tell the story of Graphic Time Two.

It's kind of interesting in that, on one hand, the cost savings really is kind of mostly I think that the CFO cares about, right?

But you can kind of package it in a way that sounds cool to engineers but then engineers are not doing it for the right reason.

They're doing it for the wrong reason. And that's kind of really frustrating.

Charity: Yeah, but I think that if they're doing it for the wrong reason but they're doing the right thing anyway like that's how most things in engineering are done is by inspiring people that show up there with the wrong reason.

Liz: So, I guess in some ways kind of these rogue observability projects where someone just hooks up open telemetry or honeycomb to an application.

Charity: That's how we get half of our customers.

Liz: Yep. That's doing it. For the wrong reason of someone has a pet project but then they get the right outcome.

Charity: People just want to be associated--

This is why Kubernetes is successful. People just want to be associated with development, right?

They want to be associated with something that is cool and hip. So they're like, "Ooh, maybe I need some honeycomb in my stack."

You know, I'll take it. You know, it's still gets them to a better place.

Like I don't really care why. Kubernetes on the other hand.

Liz: It depends whether you know, the ancillary benefits or the benefits you wind up getting in the end are actually worth it.

Thank you very much for coming on with us Corey.

Corey: Of course.

Liz: This is entertaining and wonderful as always. You're a delightful guest.

Corey: Well, thank you. I do my best.