Ep. #52, Service Level Objectives with Alex Hidalgo of Nobl9

Alex Hidalgo: So I've been doing this a long time and so unfortunately I have too many disastrous stories, I've been involved with too many incidents. Some of them short, and some of them lasted months, and you can really consider it all the same incident.

At one point in time I bought a sleeper couch just because I was getting woken up every single morning at about 3:00 AM and I didn't want to keep waking my partner up. So I actually bought a new couch to sleep on, just because I knew I was going to be woken up and didn't want everyone else to have to deal with that.

So yeah, I've had a few experiences. But perhaps the most formative was dealing with a very broken ELK stack, an elasticsearch, log stash Kibana logging system and it was something that took months and months to actually resolve. And even after figuring out what steps we'd need to take to make things better, even that took more than a month for things to actually start to resolve themselves even.

There was a process where I went from knowing absolutely nothing about this stack to accidentally becoming an elastic search expert. It was a long but very interesting experience.

Jessica Kerr: So if your logging is down, who logs the logger? How did you troubleshoot that?

Alex: That was exactly part of the problem, right? Is that--

Everybody else at the company relied on us and that actually made us, by every quantifiable measure, the busiest system at the company because if any other system, any other service was doing something, they had to let us know and we had to process all that and index it.

These were all slightly lucky in the sense that these were all physical machines, so I had access to them in a way that I was used to for my entire career.

They were Linux boxes, I was able to logon and, worst case scenario, go tail-f/var/log/whatever service I needed to look at. So while we didn't get to use our own logging system to investigate things, at least I still had access to data that way. As I mentioned, I've been doing this a while so I'm pretty comfortable with command line tools and how to parse data out of logs that way, and things like that. But yeah, it was not fun for everyone else at the company because when we were down, they couldn't troubleshoot their problems.

Charity Majors: Right. How did this change the way you saw systems, the way you did your job in general?

Alex: I think it very quickly made me really understand how important observability systems are.

Everything, your telemetry, right? Everything that tells you something about your system, if you're relying on someone else to provide that for you, that has to be priority number one.

You'd think maybe I'd know this because I was on the production monitoring team at Google, we kept all of Google's monitoring and all of Google's alerting alive. But the truth is those systems weren't that brittle, they worked fairly well, and even when they didn't, we knew the remediation paths and things like this. The problems I face with this elastic stack, no one knew what was broken. No one knew what we had to do.

Charity: You were the expert.

Alex: Well, I had to become it.

Jessica: Question. What year was this?

Alex: This would've been 2018 into 2019.

Jessica: So Elk wasn't super new, but it wasn't old hat?

Alex: No, but it was new to me because that was the first job after Google. I had spent eight years in the cathedral and I'm just coming back to what the outside world looks like, so suddenly I'm on this team and I'm the most senior engineer and they're dealing with this fire of a logging system. I was like, "Well, at least I can Google this stuff, right?" At least other people use Google products.

Jessica: Exactly, exactly. So this was like a forest fire that was like a slow spreading burn that went on for months?

Alex: Yeah.

Charity: This sounds like a good time for you to introduce yourself.

Alex: Hey, everyone. My name is Alex Hidalgo. I've been a site reliability engineer for over a decade and I wrote the SLO Book.

Charity: Yes, the SLO Book.

Jessica: What is SLO?

Alex: Service level objectives, my favorite topic.

Jessica: Yay.

Charity: Before we get to that, just one second. The most formative adage of my career I think sounds very familiar to yours. This was at Linden Lab when we tried upgrading our MySQL cluster from 4.1 to 5.0 and all of the metrics said it would be faster so we did it, and PS, you could not replicate from 5.0 to 4.1, you could only replicate from 4.1 to 5.0. So we were doing the upgrade without a safety net, right? So we upgraded and it wasn't faster for our query families and our workload.

It took us a day to realize this, we finally rolled back, we lost 24 hours worth of data from the primary MySQL, everything. I spent the rest of the next year trying to fix it. We were like, "Okay, back to square one." And I ended up writing this capture replay system, it would capture 24 hours worth of traffic from the MySQL primary and then replay it against, over and over and over. This is when I got to use the Percona patches Mark Callahan had built at Google. Should I distribute the workload over multiple CPUs? It was a heavy, heavy lift.

I learned so much from that one. It came around to Thanksgiving the next year and we were ready to make the switch, but everyone was so petrified, so terrified that they just wouldn't do it. They sat there for another two months until after January and then they finally did the upgrade and the flip over, and it was like silence, crickets. It was so smooth and so perfect, but that was my introduction to problems that you can't just fix by fighting the fires harder, right? Where you actually have to take it offline and do some science.

Alex: Yeah, totally. I like what you mentioned there about how people then got afraid, and that's real because in some senses we work pretty cushy jobs, we get paid really well, we have a lot of freedoms. But it can be traumatizing, absolutely.

Charity: It really can.

Alex: There was a time period at Google where just about every SRE had the same Android device and just about all of us used the same tone for our pagers.

Charity: Oh god, oh god, oh god.

Alex: And years later, after people had moved on from that era, we launched the big project and the dev team, the sister dev team to our SRE team, had one of those birthday cards that you open that then plays a noise. They recorded that pager sound, it was legit flashback territory because that was that sound that woke me up every day at 3:00 AM for months and months of my life. So again, I don't want to ignore the general privilege we see as tech employees a lot of the time, but that trauma can be real. It can stay with you. I don't know what I'd do if I heard that sound right now.

Charity: There are pager sounds that still give me a jolt, where I'm like, "Oh god." So yeah, I completely understand that.

Jessica: And on the dev side, what's scariest about those incidents is the organizational scarring, is that fear that suddenly we can't change anything and your software ossifies and now it's so hard to get anything out, you get it out in big bunches and then it's garbage to figure out what went wrong. Yeah.

Charity: It's the software journey death spiral, is what I've come to think of it as.

Alex: That's an interesting segue to that's how I see people do SLOs so wrong.

Jessica: Really?

Alex: They'll try to follow the original formulated pattern outlined in the first Google SRE book, which is simply, "Out of error budget: Don't ship features. Have error budget: Ship features." But I've seen this firsthand on so many teams, these calendar aligned windows, right? So they'll blow through their error budget and then the SRE team says, "No shipping features." In the meantime, the devs are still writing a bunch so then the calendar flips over and suddenly you have all the error budget back. So everything gets pushed at once, like 500 PRs, all hit prod at the same time and you instantly blow through your error budget again, and rinse and repeat. I've seen this too many times.

Jessica: So this is a total bench, paycheck arrives, buy everything, have no money, for the rest of the month.

Alex: Exactly.

Charity: So how did this inform? I'm guessing this informed your desire to work on this book, which as someone who just recently finished their second O'Reily book after three and a half years, high five.

Alex: Thank you.

Charity: You are so organized about the way you did your book. I just remember you had a Slack room, a Slack chat, you had deadlines, you had the spreadsheet. We lurched around it, we were not nearly as organized or sophisticated about it. But on the flip side, during that three and a half years, the target of observability moved and changed so much.

Alex: Cons. Yeah. I mean, same in my world. If I were to be writing the SLO Book today, there's so many things I would say differently, there's so many things I would add, so many things I'd remove. I'm still proud of it and I still think it's a useful work, of course.

Charity: Like what?

Alex: What would I change?

Charity: Yeah.

Alex: A lot more emphasis on exactly what I was just talking about, which is how to use your error budgets better, what can you do with this data. There was plenty in the book about it, but that should've been several entire chapters as opposed to sections of chapters because I think the resulting data that you get out of the SLO based approach, that data that you get that you can hopefully use to have better conversations and make better decisions and all that. Yeah, I think I could've put even more emphasis on the fact that it's not good to set strict rules around it or you end up with a situation like the one I was just describing where all you do is you stop shipping features and then you blow your budget again the next month.

Jessica: You want data informed decisions, not data driven?

Alex: Yeah, right. And I do talk about that in the book, but that's the first thing if you're asking me what I would do differently. I would frame it even stronger, that, "Make sure you're using this data correctly and you're not just setting a few policies and walking away from it."

Jessica: Yeah. Do you have data or does the data have you?

Charity: I need a sticker that says that.

Alex: New Charity sticker dropping.

Charity: Exactly. Well, so you changed jobs after this too, right? Talk to us about your new job.

Alex: Yeah. So I was a SRE forever, and then I joined Noble Mine where I'm at now, and became director of SRE for a while. That was actually my first people management role in tech.

Charity: Wow, you jumped straight to director, huh?

Alex: Yeah. It was just what was needed from me at the time, and I actually really enjoyed it. But I stepped down from that back in January after just about a year exactly. I think I was good at the people part of the people managing, but to be honest I don't think I was the best at keeping people on track.

Charity: Yeah, those are tough skills.

Alex: Yeah. I never totally internalized that, but I'm really happy with what I'm doing now. I'm the Principle Reliability Advocate and I'm really back to what I think I was originally hired to do at Noble Mine which is just be the SLO dude. I help with product, I help with marketing, I help with both prospects and signed customers and just do a little bit of everything. But I really get to take the expertise that I've been able to obtain over the last several decades and share it with people. That's my favorite, education and helping making other people's lives better.

Jessica: You get to move the world forward both internally and externally.

Charity: That sounds like what we've learned at Honeycomb too with Jess and Liz. They've gotten to the point in their careers where they're kind of odd ducks who want to a little bit of everything, all over the company. Putting them in the advocate role means that they have their ear to the ground, they can go wherever they needed or wherever they feel drawn towards. So I think it's a really interesting career path that I don't hear people talking about. You always hear about the engineer to manager pendulum. But when it comes to being a senior IC, there are other ways to be a senior IC that don't involve going deep into the Principle Engineer route. They involve more coming up, and I think that advocacy is a super interesting place for those people.

Jessica: Yeah. It's totally ironic that they call IC, Individual Contributor, where the higher you go, the less individual it is.

Charity: Totally true.

Alex: Yeah, totally. Even at my time at Squarespace I was getting to the point where I realized I wasn't the dude writing design docs anymore. I wrote them but my design docs existed so that six other people could write six other design docs. It was more high level roadmap stuff. But yeah, I like operating at this level. I think it suits me and it's fun, it's fun.

Jessica: I think that's the definition of a seniority in development and technology generally, it's how widely do you think, how widely do you influence, is it this function that you're writing, is it this service that you care for, or are you looking at the wider system throughout the whole company, and then eventually with product and stuff farther than your company.

Charity: This is also why I feel like it's hard for senior ICs to come into the company at a very high level. It's so much easier for them to work their way up, and once you've been there for a couple years, then okay, you've got the lay of the land and everyone knows you and you know everyone, and you can operate at this level. But hiring someone in fresh from outside, it's frustrating because once you reach this level in your career at a company, you don't really want to go and start from the beginning somewhere else. You want to be acknowledged for the work that you've done. But also it really can be challenging, I think, to come in at level six, seven, eight and just have that impact off the gate because it depends so much on context and relationships.

Alex: Yeah, totally. A huge part of what you do at staff+, it's the term we throw around on tech Twitter now. It's understanding the human elements of the systems, you know the history, you know who to go ask what question, you know all the different teams, you know the history of the teams, you know why this team owns that service and-

Charity: And I don't want to skip over this, it's grounded in your deep understanding of the tech, right? Because you've got project managers who know everybody and everything too, but they can't have the same influence because it's not grounded in this deep... I think it's really easy for people like us to skip over that and go like, "Oh, it's only the people stuff that matters." No, it's the engineering stuff is easy for you so you're able to leverage it into broad influence. Both parts matter a lot.

Alex: Yeah, totally.

Jessica: He's an associate technical system.

Charity: Exactly. This is why when young engineers come to me and they're thinking of skipping over to be, whether it's a manager, and by young I mean they've been in the business less than five years. I don't think you can really be senior until you've been doing something for five to seven years. Especially this happens to women because they often have, "better communication skills, better people skills," or whatever so they get pulled into being developer advocates or being manager of whatever.

And I always discourage them from doing that because technical credibility, especially if somebody is going to come up to you and question your credibility, you want to have that credibility. It's going to decay really quickly if you're not already a master of your domain, and it's not going to be as satisfying, you're not going to get as much pay. It's not going to be the same because it's not just the people stuff, it is having that technical depth and seniority and then also being good at the people stuff.

Alex: Yeah, I totally agree with you on that.

Charity: So you, Alex, one of the reasons you're my favorite people, and it's so funny that we got to know each other over a mini spat about your book where I was like, "Eh, I don't like this." And you're like, "That's fine, it's not for you." And I'm like, "Fair enough." But one of the reasons I love you so much is because you're like me, you're not a dropout like I am but you were a liberal arts major and you said you studied history and philosophy. Tell us how that's really informed your tech career and why you think that's a good path to a job?

Alex: Not just what I studied in college, but the whole route of how I ended up in tech in the first place, because I spent my 20s not just studying history and philosophy, but working at restaurants. Both on the line in the kitchen, as well as front of house and bartending, and I worked at a furniture store for a while.

Charity: Where?

Alex: I worked in the warehouse, sometimes I got to drive the big truck and deliver furniture. I've done a ton of stuff, but really all those experiences have informed how I approach everything in tech. Everything is a people problem in the service industry. Everything. Because there's always customers involved and there's always your teammates that you have to rely on. You cannot work in a kitchen unless everyone's on the same page, and I could see so many parallels from that, that I think otherwise, especially people who study computer science and go straight into the industry, it can be difficult to learn those skills as easily in tech.

I think people do eventually, but it's not as clear and it's definitely not the case that your Comp Sci education necessarily prepares you for that, I think. So I've been informed by lots of different stuff, but it's one of the reasons I've come to love SLOs, to come back to them again. It's difficult to talk to me without me bringing those up all the time.

But I realized it was stuff that I was already doing in all my previous careers. When I was a bartender, I would have this objective to greet every customer within 30 seconds, and I didn't have a stopwatch necessarily.

But I was keeping track as people walked in, "Okay, I got this person, now I got to get that person," because I wanted them to have a good experience, I wanted to provide them with the proper level of service. I knew if I was able to hit those targets often enough I'd have a good night and I'd make a lot of tips and everyone would be happy. But if I missed those targets too often, I'd have a bad night and I wouldn't make as much in tips and people would walk out and maybe not come back. That's all service level objectives are, and I realized I'd done them, every career I'd ever had. We just didn't call them that.

Charity: I have a sticker that says, "SLOs are the API for engineering teams."

Alex: Yeah. They're a communication tool as much as anything else. They're a way to interface with people as much as they are anything else.

Charity: Because if you haven't set these agreements with your neighboring teams, then you have to know everything about what's going on and the nitty gritty or whatever. If you've made these agreements then you don't have to be cognizant of what's going on underneath the hood, as long as they meet their agreements, it's up to them and it's up to you, as long as you meet your agreements. That's the only way to scale teams.

Jessica: So it's like setting good boundaries?

Charity: Yeah. It's very much like setting good boundaries in a relationship, yeah. Talk to us, you're the only one of us here to have an actual CS degree. Did that make you super powered?

Jessica: It's not a CS degree though. It was physics.

Charity: Of course. I forgot.

Jessica: But I observe that people totally accept that.

Charity: Oh yeah, of course. Because it's a, 'Hard Science.'

Jessica: Right, right. Physics is supposed to be harder than Comp Sci. The math is important, the lab stuff, the scientific method of varying one thing and see the difference, oh my gosh, so useful in debugging. Not everybody has that drilled into them, totally useful.

Charity: Well, how did you get into computers, Jess?

Jessica: Let's see, I was programming my calculator in high school for lack of anything better to do, and then in college I got really lucky and my aunt knew somebody at Fed Ex who knew somebody in operations research who got me an internship. I was like, "This is great, because I get to do cool stuff and solve puzzles and go home at 5:30 and not think about it anymore. There's no homework, it's amazing." Of course now it's nothing like that.

Charity: You've had such an interesting career, Jess, because I feel like you've had almost a renaissance career after because you were in big companies doing backend stuff in Java programming for a long time.

Jessica: Enterprise Java dev, yeah.

Charity: Enterprise Java dev, and then you discovered conferences, is how it goes in my mind.

Jessica: Totally. And then it got really interesting, and suddenly software was fascinating because when you meet other people who are fascinated by it... This was at the renaissance of functional programming and mobile and a bunch of other things. Yeah, so I got lucky and then you go to a conference and you listen to some talks and you get ideas for the next things you want to talk about and so on.

Charity: Yeah. This is how extroverts go about things, I've learned. Well, you were saying earlier that the humanities classes really stuck with you. How so?

Jessica: It was engineering school, there were like three humanities classes, a semester of French, a semester of what was supposed to be philosophy, but everybody in physics of course just took logic which was child's play. But it's useful. Then there was one English class where we read important things about the Death Marches of prisoners of war in the Philippines in World War Two, the first Lovecraft I ever read was in that class. And now, what do I read in my spare time? It's philosophy. I'm trying to go back and catch up on who are these people that everyone is talking about, and it's fun how much that comes back to software. Alex, how have you found your history and philosophy background?

Alex: So I majored in philosophy, I minored in history and I almost minored in English, but I had like one more senior seminar to take and just didn't care to finish it up right. But I almost had two minors, and I like to joke that I took the three most useless degrees and combined them into one. But those are probably the three degrees that require the most reading and writing, that's all you do. At one point in time, one of my last semesters, I remember I had something like 120 pages worth of papers, all due within a week or two and just-

Jessica: No wonder you can write a book.

Alex: Well, that's the point. I learned how to synthesize data, I learned very well, I think, how to take complicated concepts, distill them into simple ones, explain them to people. All the reading and writing I did I think really helped prepare me for my tech career more than-

Jessica: Being a good writer is such an underrated skill.

Alex: Yeah, totally. It has consistently helped me out in my career. Everything from being able to compose a good email to explaining things correctly on Slack, knowing what questions to ask. That's absolutely what I took away the most, was the ability to communicate, the ability to absorb information and then the ability to share that information out.

Jessica: Well, it helps in code too.

Charity: I've always been a good writer, I was homeschooled so I never really had school. But I did a lot of reading and I did a fair amount of writing. What helped me in my career the most was getting good at public speaking because people... I think I play an extrovert reasonably well on TV so people don't always realize this, but when I was at Parsed, this was 10, 12 years ago now, I remember the first talk I ever had to give to the company which was 12 people at the time.

I was supposed to give a talk about the infrastructure, I was so fucking scared. It took me two weeks to get up the nerve, I wrote the entire script verbatim and I stressed about it, I had nightmares, I woke up in a cold sweat day after day for like two weeks. Then when I gave it, I was shaking, I couldn't function, I couldn't talk or think while people were looking at me.

I was so humiliated by that that I just leaned the fuck into it and I started speaking everywhere that would have me. Now that I know ADHD and adrenaline is a stimulant that I really get off on, but I could never have started a company, I could never have done the things that I do now if I hadn't pushed through that fear and become, not great at it, but good enough at it.

Alex: Totally. I too play an extrovert on TV. People always think I'm extroverted, in fact I thought I was an extrovert up until a few years ago when my therapist was like, "What're you talking about?" "Well, what do you mean? I'm good at talking to people and I'm extroverted, right?" She said, "Alex, how do you get your energy?"I'm like, "Going on long walks by myself, listening to podcasts." And she's like, "What do you do after you have a board game night with your friends?" "I have to go spend time alone, I have to."

That's how I actually recharge, is being by myself, and it wasn't until I was 37-- I spent my entire life assuming I was an extrovert and then a professional explained to me I just wasn't, and that helped me understand so much better how to engage with my public speaking, for example.

I was able to better recharge and therefore better approach the scenario and therefore it wasn't as scary anymore.

Charity: How has the pandemic impacted you?

Jessica: Yeah, how was it when you weren't getting that socialization?

Alex: It's been in some sense, I think, easier for me than some people because, again, I am so happy just being by myself or just spending time with my dog or my partner and going on very long walks. For a while that was the only thing we could do, we had nothing else to do. It was sit at home or go for a walk, and luckily that's stuff that I truly, honestly enjoy. But on the same token, I get so much out of doing the public speaking stuff. I am so sick of the virtual conference thing. I'm helping to organize one right now. I don't know when this episode goes up, but go register for SLOConf, SLOConf.com. It may not have happened yet, I don't know. But personally, I just can't wait to get back to being on stage, in front of a room. It's the performance that I miss.

Charity: I was asking because for me, I feel like I spent the past decade and a half of my life in training like a marathoner, being around people, just pushing myself constantly to push my social boundaries, be around people more. Because it used to be when I was working at Linden Labs, I could go out for drinks with folks maybe one night a week and that'd be it. I could last maybe 30 minutes, maybe an hour or two, and then I would shut down for the week and that's not okay. Not okay with me, so I was pushing.

Now I feel like after two years of enforced isolation, I'm having a really hard time opening back up. I'm having a really hard time. It'll take me a couple of hours to force myself out of the door to go visit friends, and then I have a great time, but also after 20 minutes to an hour, I am tanked and I start getting really irritable. I'm having to pay really close attention to my feelings and my ability to be around people. It has lessened, but it's also very unpredictable for me and it's just really weird right now.

Alex: It's not like we have to be the same people we were before. There's not some kind of recess. If I look at how different I am in any two year period over the entire course of my life, I have become a very different person. And so it'd be silly to be like just because we had the pandemic, and hopefully it'll, 'end at some point', that things are going to be... No, they're never going to be the same. We've all been changed by this, not just because it's been a collectively traumatizing experience. It's two years have passed, of course we're all different people.

Charity: I don't like to hear this one bit.

Alex: I'm sorry, Charity.

Jessica: What makes you isn't how you are at any one moment, it's that process of change, it's the delta. It's how you direct your attention, and therefore the shift in your direction and your future self. It's how Charity leans into what scares her that makes her Charity. Not being shy or being able to present, neither of those things is the delta.

Charity: Yeah. I guess it scares me because I'm starting to sign back up, I'm going to be in New York next week and I'm going to Japan next month and stuff. It's like I feel unsure in my ability to make commitments because I'm unsure how much bandwidth I'm actually going to have. I've been to so many conferences in the past where I ended up spending half of it in the bathroom just hiding from people, and then I feel really sheepish and dumb.

Jessica: Only half? That's pretty good.

Charity: Thank you. Then I feel like it's a waste of the company's money and everything, then I excuse myself and I'm over it. But you're right, you're both right. Of course you're right.

Alex: I was someone before the pandemic, I already didn't love crowds. Before the pandemic I already was someone who I'd stand on the subway platform here in New York for 20 minutes, waiting for the empty subway to show up. Easily. I don't know how I'm ever going to go back to rush hour subways or going to a busy bar or a club. I think I'm probably done with that for life at this point.

Charity: Jess is lusting in her eyes right now, just like, "Busy clubs."

Jessica: During the pandemic I had dreams about being on a crowded subway, good dreams.

Charity: Wow. Here's to getting all of our needs met, whatever they may be. The last thing I wanted to talk about was related to what we were talking about, about the CS degrees and whatnot. But I feel like there are very few junior SREs in the world, I feel like so many of us from the ops side came from very non traditional backgrounds and the bar was so low for getting into the tech industry.

It was like, "Do you know the command line? Can you name these two flags to graph? All right. Here you go, cowboy. Saddle up." And they just toss you into the deep end. Now there's been this whole professionalization and training people into engineers. On the one hand I understand why and I support it, and on the other hand, it's like how do we get people in who understand systems, not just software? Where are they going to come from?

Jessica: Do you need five or ten years of some shit or other before you're ready to-

Charity: No, I didn't have five or ten years. I went straight from being a dropout music major to admining the CS department's servers and the maths department. I went straight into being an ops person, but I didn't have a CS degree. Now if you want an entry level position, you better have a degree of some sort or come from a couple of years of Hacker Academy or something. Even then, it is so hard to get your first job in engineering.

Jessica: Yeah, it's totally a problem because everybody needs the experience but-

Charity: But nobody is willing to pay for it.

Alex: Yeah. It's tough. I don't know if I have good answers there, but one thing I will say is there is a huge number of people I look up to came the IT route. That's how I got here. I was desktop support at some point. My first job when I moved to New York was traveling around to various client's offices running virus scans and stuff. And yeah, computers were also a hobby and I had been coding since I was nine and messing around with things and all that. So I had this other background as well, but that's, I think, a great pipeline. Let's look to IT, let's look to desktop support. I think those are the kind of people we can level up.

Charity: Yeah, that makes sense to me.

Jessica: Yeah, because they already have troubleshooting skills and dealing with customer stuff.

Charity: But I don't encounter those people. Only when I worked at Facebook did I encounter any of these people.

Jessica: You mean help desk support? But we have customer support people.

Charity: We have a couple of customer support people, yeah. We've explicitly been thinking about this as a route into tech because we have tried hiring a couple of junior engineers and it's tough. I talk snark about how nobody wants to invest in it but everybody wants to reap the skills of it, but also we haven't unlocked the secret to doing this. Our success in bringing juniors on has been very, very spotty because... especially with the remote teams now, how do you bring on newbies with remote teams?

I think it's really hard because so much of them, there's a lot of anxiety and nervousness and they need somebody to pat them on the shoulder and look over their shoulder. Really it's best to see if they seem like they've stalled or do they need my help? Part of it is just being as early as we are, and as strapped as we are. Every single person we have has a full time job and we can't spare a third of someone's time to coach someone, even though in the end that's what it takes.

Alex: Right. You want to be the kind of organization that says, "We can mentor and we can bring on newer people to the industry and help them learn, encourage and coach." But the reality is, how many of us actually have the cycles to do that? And if you're not being honest with yourself, then you are both letting yourself down and you're letting that person down.

Charity: Something that we've found though is that it isn't really the most senior people who are best at this. It's often intermediate people, because they remember.

Jessica: Yeah. The experts are not the best teachers.

Charity: Correct. It's people who were junior engineers in the last three to five years, they can help juniors get into tech.

Alex: Right, because they remember what it was like to learn the things.

Jessica: And also it was so different when I started.

Alex: Oh god, a little while back I was talking to someone and they were like, "So when we move to IPv6, are all of our ports going to change?" "No, it's IP that's changing. Not TCP." They were like, "What?" Because they didn't even understand that TCP and IP are entirely separate protocols and blah, blah, blah. But you had to know that if you were of a certain age because you had to deal with networks.

But there are so many people in the industry now, even people who've been in the industry for five or ten years who've only ever run on the cloud, have never had to think of networking outside of what IP does my Qube pod have? If they even have to think about that, because they have some kind of service mesh that actually handles all that. It's little examples of things, but there's stuff like that that some people just don't learn anymore because they don't have to.

Jessica: And yet they have to learn so many more things because they have to know the service mesh and the cloud, and there's like 16 layers on top of that. So yeah, you know fewer of the lower ones, but we just didn't have so many things to learn.

Charity: We had different things to learn.

Alex: Right, different. But to this day I could probably teach you a lot more about networking than I can about any service meshed software I've used.

Charity: For sure.

Jessica: Yeah, because it was like with that elastic search that you got super expert in. You had that expertise earned in the fire.

Alex: Right, yeah.

Charity: And when you know the fundamentals, it's so much easier to figure out things that are above it. If you didn't know the low level networking and file system and everything, becoming an expert in elastic search would've been way harder if not impossible.

Alex: Yes, especially again because these were a bunch of physical machines, in a data center, with actual hardware techs that worked for my company. Yeah, even though it was only a few years ago, it was a much more 'old skool' experience. But I had done that.

At some hypothetical pure level, if you want to learn computers, I feel like you've got to learn how to count in binary first and then learn what a logic gate is. But no, I'm so dumb, that's not what you actually have to know anymore.

I think it may have been a good starting point 20 years ago, and part of me still thinks people should know how to count in binary. But you don't have to know that stuff anymore, you really don't.

Jessica: Well, it doesn't hurt. You could do a couple of puzzles in binary. Spend a class period on it. That's enough.

Alex: Yeah. That's fair, that's fair.

Charity: You need to be able to recognize powers of two, because you'll be screwed if you can't do that.

Jessica: Yeah, that should probably be a useful conference talk of just what is binary for the people who never had to learn it.

Alex: I developed a networking course for Coursera a few years ago and the two largest sections are on subnetting and DNS. And people were like, "You really had to spend this much time on subnet math?" I'm like, "Yes, because this is what I've seen cause people the most problems and break systems the most often, and DNS being right up there as well." I was like, "No, we are covering this because is what people really need to know."

Charity: I could not agree more.

Jessica: Yeah. We can wish it was implemented in a way that was easier to understand, but the world is what it is, and this is now the physics of the internet.

Charity: Well, it is what it is and we all still use SMTP, and we always will until we die. And on that note, it's been great having you, Alex.

Alex: Thanks so much for having me. It was a lot of fun.

Charity: All right, cheers.