Ep. #12, High Score Bills with Corey Quinn of The Duckbill Group

Heidi Waterhouse: So, today on Unintended Consequences, we are talking to cloud economist Corey Quinn.

Hey Corey.

Corey Quinn: Hello. Thank you as always for indulging my ongoing love affair with the sound of my own voice.

Heidi: I am a developer advocate. If I didn't have that same muse, I don't think I'd be in this position.

Today, we're going to talk a little bit with you about what an unintended consequence is and how you see it from your point of view, as somebody who battles AWS bills.

Because I feel like a lot of people, even very expert people. Hi, Chris Short, I love you, I'm going to use you as an example.

Even very expert people get caught in AWS billing and that definitely seems like an unintended consequence to me. So how does that happen?

Corey: Well, my entire career has been built on unintended consequences.

Let's be clear, if you had asked me five years ago, when as I recall you were the first guest on Screaming in the Cloud, and I just started out my nonsense, "So what is your plan here?"

"Oh, I intend to start aggressive shit-posting about cloud computing and its bills. And in turn, turn myself into the closest thing that this corner of the industry has to a tech celebrity, quote, unquote," you would've looked at me like I was nuts because it sounded nuts.

I had no plan on that, this is an accident.

So again, my entire life is an unintended consequence in one direction or another.

And you're right, that manifests as well in cloud bills too.

No one sets out trying to get the high score on the game called AWS billing, but they do sometimes.

And it leads back to a whole bunch of different things.

Cloud computing is one of those powerful things that lets you do such a tremendous number of things and bill you in perpetuity. Turn something on, experiment, great. The cost of that experiment was something like what? 17 cents. Forget to turn it off, great, it's going to keep billing you in perpetuity until the Earth crashes into the sun. And that becomes a problem. You're not billed for what you use, you're billed for what you forget to turn off. And that manifests itself very differently, depending entirely on who you are in this conversation.

Kim Harrison: I remember recently there was a college student who probably did something like this, turned a thing on, forgot about it, came back a couple days later and found themselves with a massive bill.

I think you even commented on this.

There's some Twitter thread about like, "Oh God, someone please help. There's no way I can pay for this. How did I get here? And how do I undo it?"

Corey: That's the best part, is that you don't.

You just wind up begging mercy when you wind up with a high score bill.

And it's fun and there's an entire world, or there's an entire ecosystem, let's call it, of people who love to play the told you so game of, "Oh, you wound up not turning it off? You should have read the terms and conditions of the free tier," and you don't work for Amazon, but you sure sound like you.

I assure you, there are billing dimensions that catch everyone by surprise.

If that weren't true, I don't think I would have a business.

It's fun and it's exciting, but it's also the lack of understanding that often permeates the industry, is that I don't actually cut AWS bills. That's a byproduct.

What I actually do is help companies understand them in the context of their constraints, their strategy, where they're attempting to go and make them more predictable and validate on some level what they're spending is the right amount.

When you're a giant company and you have a whole bunch of users all over the place and you're spending, I don't know, $60,000 a month on providing that service.

Is that the right amount? In some cases, no, it should be twice that, because you want to have backups occasionally, or if that one server goes away, so does your business.

There's an idea of resiliency and trade offs here, but at a certain point, cloud costs and cloud architecture are indistinguishable from one another.

Heidi: That's a really good point. Let's back up a bit. Kim is visiting her parents who are computer programmers and they have a question.

Corey: Oh, wonderful. I love providing tech support for people all the time.

You think I'm kidding, I'm not because I love fielding questions from people who are outside of our space, just because it lets you get a healthy reminder of just how esoteric and niche what we do really is. Please, ask.

Kim: Well, this room that I'm in, which our listeners can't really see.

Picture of the Beatles, I've got a Venetian flag over here. I'm in my mom's office.

My parents are programmers, but I think the height of her career was the '90s.

She worked for NASA, she worked for the Hubble Space Telescope. My father did a lot of defense contracting work.

They come from the days of waterfall and build a thing on this disc and deliver it.

So at some point my mom retired, probably I want to say, '05 or '08.

She asked me, "What is the cloud? I keep seeing things. What is this? I don't understand."

When I explained it to her, she seemed a little nonplussed like, "You've got to be kidding me, that's it?"

I mean, it's just using other people's computers. So is it more than that?

I mean, is the cloud that simple and marketers love to talk about it in special ways to make it sound more interesting or is it really that simple?

And it opens us up to more possibility, but what is the cloud? How would you describe that?

Corey: The answer is, as always, when you get into the edges or the margins of anything is the same.

It depends because there is truth to what you say, but it manifest the way that folks often think it does.

Sure, it's just someone else's computers that they have invested tens or hundreds of billions in research and development into not just the computers themselves, but the services that are running on those computers and how they manage those computers and how they account for failure of those computers without breaking things and I assure you, unless you are yourself, a hyperscale provider at that level, you will not do a better job of running the computers than the cloud provider will.

Now, viewing it as just a place to run a bunch of VMs, that is an approach to cloud, generally not one that's recommended, but it can be done, but the value also lines and higher level differentiated services.

That adds significant differentiated value.

It does require, on some level, picking a horse at some point to ride rather than, "Oh, we'll use all the clouds, because that seems like the best option," but it does lead down interesting paths.

But yes, fundamentally under the hood and all the services, et cetera, et cetera, cetera, it is someone else's computer.

The billing model is psychotic and there's a bunch of different ways it manifests.

And what you do on the computer is all well and good.

But by that same token, everyone who works on computers for a living from certain point of view in a similar, simplistic way, our entire careers are spent doing exactly one thing and that is making the pattern of lights on the thing in front of us form different patterns.

If that isn't a depressing way to think about it.

Heidi: We tricked Sand into writing poetry and it was all downhill from there.

Corey: Yes. And then, oh wow. Some of the poetry it's writing is profoundly racist.

How do we fix that? And it turns out that bias laundering isn't really a thing.

Heidi: I think that's an interesting answer. So you said differentiation.

What did AWS or Google figure out they had to provide once they started going hyperscale?

What do they need to give us to make it not just VMs that somebody else hosts?

Corey: Initially nothing, that's the best part.

But then they start looking at what customers are doing and it's so, "What are you working on today, buddy?"

The answer is I'm trying to figure out how to build a load balancer out of HAProxy.

And at some point, a lot of people are building load balancers and unless you're F5, maybe building load balancers isn't what you should be doing for a living.

So, okay, we start seeing these global problems that people are solving locally. I've always been a fan of not doing that.

When I have a problem that I'm relatively sure no one has discovered before or had to solve, before I start writing code, which no one wants to see, I check GitHub or sorry, GitHub, which is I believe how it's pronounced.

I hope that someone has taken a whack at this before, because I assure you I'm not going to be able to reinvent something this weekend from first principles that's better than Terraform, for example.

Same story with feature flags, I'm going to talk to LaunchDarkly before I wind up half-assing something else because you folks put your whole into it and generally leads to better outcomes.

I will bias for buying something before building something if it needs to work in a serious context.

If it's my weekend shit-post that I'm building myself and spare time, labor of love, okay, different story.

But when you're doing something professionally, your time is as far from free as it can get for your employer.

So, go ahead and look at buying things, that seems to make sense.

That's what differentiation comes down to as, in some level, you're going to be paying one way or the other, may as well move up the stack of it to have a managed database provided for you so you don't spend your days configuring MySQL yet again, or PostgreSQL, or whatever your database of choice is.

Heidi: So, what you're saying is that the differentiated services are in fact, an unintended consequence of giving them access to what we're trying to do?

Corey: On some level yes, but they also have run these things themselves before internally.

I assure you, someone at Amazon, long before Amazon Web Services was ever a thing was sitting there going, "How do we build a load balancer? Because it turns out that one computer isn't working for us anymore."

That is a common evolution. The folks building these things have been solving these problems themselves.

At some point it just gets ridiculous, where Ground Station is an Amazon Web Services service.

Yes, services service is a thing that happens. What is the problem we're solving?

How to talk satellites in space. I don't know that I've ever worked at NASA like your mom did Kim, but I've never had that problem, how do I talk to satellites in space?

I've had other problems, like how do I keep people from large cloud providers from threatening my life?

Heidi: Oh God, that's terrible. Well, if there were money in it, they'd probably offer it at a service, so sorry.

Corey: Oh absolutely, threatening Corey sounds like a sponsorship option that you'd think they would start offering.

Heidi: So, everybody has been talking about cloud native. What is cloud native? Is that just-

Corey: A marketing term that does very little there, there.

You have companies describe themselves as, "Oh we are cloud native," but they're drinking data center borscht and talking about the old architecture with a thick, heavy data centery accent and no, you're cloud immigrants.

Let's be clear here. We're cloud nativism doesn't give rise to anything positive and I wish companies would learn that a bit more effectively but here we are in the fullness of time.

The painful part that I keep smacking into with all of this is cloud native aligns with ... What is cloud native?

It is whatever aligns with what I am attempting to sell you right now.

I'm not a big fan of that because you can describe an architecture on a whiteboard and make it look amazing.

But here in the real world, we have to deal with legacy.

Legacy is of course, a condescending engineering term for it makes money.

And you want to have it continue to do that.

You can't take your mainframe and shove it into the cloud as is until they release the AWS 400.

I don't think that's on their roadmap this year at Reinvent, but it wouldn't be the first time that I've been surprised.

Heidi: I think they would make so much money if they could.

Corey: Oh, absolutely.

It's a, "If you want to wind up moving your workload just as it is to the cloud, on the one hand, you really shouldn't do that on the other, if we tell you that you're not going to give us as much money."

And Amazon's religion is of course, money.

Heidi: So speaking of Reinvent, what does it matter?

Thousands of people are going to come share germs, which is especially trippy this year in Vegas and for what?

What is the advantage? Why do they do this?

Corey: Who are these people is part of the question I'll be there because it's the center of the AWS universe.

And well, if Amazon wants to kill me, a virus is a decent attempt at it. Go for it.

I'll be self-quarantining at a hotel in San Francisco for a week afterwards, also lying flat on my back because Reinvent is doesn't make you feel great the week after, at the best of times.

For folks who are sponsoring the expo hall there, they wind up getting a whole bunch of leads, of people wandering through.

I think that's generally the wrong direction, that's why I have my Sidecar virtual event, re:Quinnvent, which is like Reinvent only good.

So, that's fun. It distills it down, it has a sense of humor and a personality that isn't written entirely in corporate speak.

It distills down two hour keynotes into things like, "Here's 10 minutes of me explaining what was released while simultaneously making fun of them."

It's fun and it's enjoyable.

There is a constituency also that gets overlooked, which is AWS employees, themselves. An awful lot of folks have spent blood, sweat, and tears building out new and exciting things and this is where they get their moment of the sun to see these things unveiled.

Sure, it's not their fault that last minute they're given a horrifying name for the service, but that's okay.

The thing that they've built is genuinely good, aimed at solving hard customer problems and the moment, it's a big reveal for them, where they finally get to tell their friends and family they've been working on.

That's not nothing. This all stems on some level from yesteryear, when the releases were nowhere near as massively prolific as they are now.

But now it's one of those, all right, Adam Selipsky's going to get on stage and effectively fire hose releases at you for two and a half hours and okay, I'm sure some of those things are going to solve my problem, but honestly, I'm so overwhelmed.

I can't keep track of it all. That's why I have a newsletter in the first place.

I wish Amazon could learn that it is okay, in fact, to release new services outside of a single week in December.

Heidi: Wait, they release everything that week?

Corey: Almost. They released feature enhancements and neat throughout the year on a constant ongoing drumbeat basis.

Once in a while they'll release a new service.

Sometimes it's great, sometimes it's awful, usually it's somewhere in the middle, but they're sporadic.

You don't see, "Launching, new, a brand new service," more than a few times a year except at Reinvent where they're all dragged out on stage.

Then of course, you also talk to companies whose marketing departments fail to really get it and, "We're doing a big feature announcement at Reinvent."

With all the struggles to split attention, the only thing you should be announcing at Reinvent is your data breach, unless you're doing it on stage during one of those keynotes as a partner thing.

Which, okay, great, go ahead and do that, but it's not the time to talk about your feature enhancement or announcement, because people are already struggling with information overload.

Heidi: That's a really good point, that we go and fill our brains for a week and then have brain indigestion and it's not a useful time for people to be adding anything new.

Kim: Right before they disappear for the holidays too.

Heidi: Right, yeah. You get back, you have con crud, oops, it's Christmas.

Corey: At some point you're lying flat on your back trying to remember how this stuff worked and it's hard to get in touch with people.

I mostly plan for the second half of December and most of January to be dead in my market.

It's sad because there's a whole bunch of videos, there's virtual Reinvent, we're going to live stream keynotes.

Good for you, welcome to 1996. And they're going to further, put up all the breakout sessions on YouTube.

Yeah, there's 1,500 videos that get dumped. Paradox of choice is a real thing.

Why not revisit it? Why not MST3K one of your own talks and make fun of yourself as you give the talk and do that as an event on Twitch or something?

Because that would take creativity and looking backwards and we're forward looking here.

You always feel like you're missing so much of the content there.

That's what I'm trying to solve for. If people say, "Well, you didn't talk about my talk or my video."

It's yeah, get in line. There's a whole bunch I didn't.

Heidi: Yeah, it's an interesting problem.

And as a sponsor, part of the reason we're going is to be seen, it's sort of like an Easter hat thing.

If you don't show up at Reinvent, is everything okay?

Do you have enough money to pay homage to this big event?

Corey: Yeah. I'm not AWS partner for a variety of conflict of interest reasons, but you need to be in order to officially sponsor Reinvent.

So I do a whole Sidecar nonsense thing and when I talk to people about sponsoring my nonsense, the response is, "Wait, so let me get this straight. For sponsorships, you're basically offering to make fun of us."

Then it goes in one of two ways, either A, "Are you out of your mind?"

Versus the, "Oh, how much does it cost?"

Because there's a misunderstanding in some respects, that no one with two neurons to bang together in the audience is going to not do business with you because I make fun of something about your company.

But there's a very real chance that the answer's going to be, "Yeah, yeah, ignore the fact that the clown is making fun. I'd never heard of them before. What they're talking about sounds like a problem that I'm having, maybe they can solve it for me."

The number of returning sponsors I have including LaunchDarkly, demonstrates that this is in fact working.

Either that, or it means a whole bunch of companies have no idea how marketing works and they like my presence in the ecosystem.

I assure you, no one has ever said that.

For better or worse, contrary to popular belief, companies do not make a habit of just throwing money away for no return on a recurring basis.

Heidi: More or less, that's true.

I'd like to believe that's true all the time, but I have worked for companies.

Corey: There are exceptions.

Let's be very clear on this, but again, I've been doing of litany throughout the year and every time one of our sponsors raises a big round or gets acquired, I have course claim complete credit for it on Twitter, because as one should.

Heidi: Because marketing.

Kim: It's an honor. If we're on your radar, it feels good.

Corey: Oh yeah.

What's always fun is watching folks who have wound up asking about sponsorships and we never actually get it across the line then they have a big announcement and it's one of those, can't claim credit for that.

They're dead to me.

Heidi: Spite-driven marketing.

Corey: I am small and petty and that is something I lean into sometimes.

Kim: When you think about Reinvent, how do you choose?

Corey: I wake up screaming. Sorry, what is your question?

Kim: As do we all. How do you choose what to focus on?

I mean, is there a rhyme to your reason or every year, it's just kind of like what catches my eye and what do I feel like?

I mean, how do you think about this when you go into it?

Corey: Increasingly it is becoming more structured for me.

The reason behind that is that for better worse, I have achieved a goal of people know my name and that means that I'm not having to do what I did at my first Reinvent, which is look around frantically to find a friend, because I don't know anyone in this whole ecosystem.

I don't understand what's going on around me. Instead, I plan my days out, I'm doing a lot of video work and of course constant live tweeting of nonsense.

For the most part in the in-person Reinvent, I mean I'm guessing, the last time I was at Reinvent, my audience was less than a third of what it is now, but the keynotes are a give me.

Making fun of the keynotes through a live tweet thread is a staple of mine and keeping up with what's interesting, what's been announced.

I sometimes have a variety of meetings to go to on behalf of some of my clients, which I don't know how many of those there'll be this year because let's face it, who is a company sending to Reinvent?

The answer often distills down this year to who is the most expendable person we have?

Often that person does not have deal negotiation or signing authority, so we're going to see how this plays out.

Heidi: That is a good point.

Corey: Then as far as breakout sessions, super easy for me, I almost never attend them.

The reason is, is that one, I'd have to stand in line for an awful lot of them and try to get in which okay, I get it, but that's a lot of time I could spend doing something else.

The whole thing is being recorded as going to be on YouTube later and I lie to myself and to convince myself, I'll watch it later and then I don't.

The exceptions to that are usually when I have a block of time that didn't fill up with meetings, which is rare, and I know the speaker and oh yeah, I'm right near there, I will drop in and watch their talk and be supportive, especially new speakers, friends of mine, et cetera.

I make it a point for that. But the demands on my time are sometimes intense and I usually don't go to too many conference talks that are breakout sessions, just because there's so much else that's going on.

My entire relationship with conferences has shifted dramatically from the first time I went to a conference I think in 2009.

Kim: Okay, slightly selfish question here.

Over the past year or two, a lot of conferences have been putting their material online just because we can't hang out in real life.

Are you actually watching any of those?

Corey: A few of them. Well, to be clear, let me start off here with the honest answer that I am not the typical consumer of information.

I don't watch a lot of videos and that includes movies and TV shows as well.

I've talked about this a few times in various places, but I have some really weird expressions of ADHD.

It's not that I can't pay attention to these things, in so much as they're not fast-paced enough for me to absorb the information.

A conversation like this is fine because interactive and it's fun and we're having a conversation.

I sit down and watch something for 45 minutes, I can read an awful lot of material in 45 minutes and I absorb information better that way.

I have the same problem with listening to podcasts, which is a weird thing to say as a podcaster.

But it's also why every episode of Screaming in the Cloud has a full transcript and does going back to the very beginning.

Kim: So, if somebody were to share a talk and they did so with a transcript, it might make it more accessible to you?

Corey: Potentially. Again, I'm not your target market for a lot of this is.

"Hey Corey, do you want to watch this thing?"

And my response is, "And then make fun of it? I mean, what is the goal here?"

I don't get those requests nearly as often as you might think.

Heidi: I find that surprising.

Corey: Well, the challenge too is I always wanted to put together an offering of, I'm going to MST3K a talk where I basically yell commentary over it as it goes.

The problem is it works super well when you think about it for big corporate keynotes, where they're announcing things like at Reinvent, and it completely falls to pieces as soon as you start taking it one step beyond that.

Because, okay great, it's funny.

When I get up there and make fun of a torrent of Amazon announcements or a bunch of Google announcements or a bunch of Azure, pretending that they're a real cloud, that isn't rife with security problems everywhere, that's great.

But as soon as it gets down to a breakout session somewhere where it's someone who works in developer advocacy or whatnot, then it turns into, I'm making fun of a person and their work.

And given the power imbalance, I can't even necessarily ask them directly, "Hey, is it okay if I make fun of your talk?"

Because it's hard to say no for some folks, when someone who with my perceived position winds up making that request, so it doesn't work.

The only talks I can really just get up there and completely tear to pieces beyond corporate keynotes are my own talks.

Heidi: I've thought about doing MST3K style five year ago talks and be like, "I was wrong here and here, oh dear."

But I haven't made the time to do that, I think it'd be fun.

Corey: It really could be. The challenge of course, is what is the forum for that? Is it a live stream? Is it a recorded video?

So now watch the recorded video of the recorded video. It starts to get a little weird.

Heidi: I think Twitch is the right, live interactive way to address that, but I would have to set that up.

Corey: Yeah, computers are super annoying.