Ep. #8, AI and the Future of Debugging with Dani Grant of Jam

Eden Full Goh: Hi, Dani. Thanks for joining me here on the How It's Tested podcast.

Dani Grant: Thank you so much for having me. Excited to be here.

Eden: Yeah. I would love to hear a little bit more about your company, Jam.Dev, and also just very curious about your founding story. I think what's really cool is you and I have a lot in common. We both started companies inspired by our experiences in testing, working as product managers and I'm curious to compare notes today and just see some of the similarities and differences.

I think what's really special is Jam is a product that's designed to make web bug reporting a lot easier and Mobot is something intended to make mobile experiences easier. So yeah, just curious to compare notes today.

Dani: Yeah. It's amazing how much testing product managers do at most companies. Of course, there's a huge role in testing for QA. Some companies like Cloudflare where I met my co founder and where the story of Jam starts, did not have a QA team. They were just to early, and so product was QA, and so we really got to see what the process looked like.

But even on teams where QA and product can partner together, product managers are just responsible for ensuring the user experience, they do a ton of testing. To this day, a third of our user base is product managers, so it's a testing product.

Eden: That's interesting. It sounds like you stumbled upon this problem that you wanted to solve while working at Cloudflare, so of all the different sort of, I guess, experiences that you had as a product manager, what particularly stood out about bug reporting and QA in particular?

From Product Management to QA

Dani: At Cloudflare I was so lucky to join this growing, early, fast moving team. I joined as the fourth product manager in the company and was there through 1,000 people. My co founder was there through IPO, and it was there that we were put on this really exciting team.

The company wanted to take big risks, find new, greenfield opportunities, but didn't want to distract the whole company from it. So they created this side team that was a sort of R&D lab to go and take big bets, try big things. My co founder and I were the two product managers on that team.

We worked for 30 engineers and our job was to launch net new businesses for the company, and if they worked then, great, we would hand them over for someone to run them in the long term. If they didn't then, okay, we censored them quickly and move on.

We needed to move fast. We needed to ship things that worked great out of the gate, but we found over and over and over again, the thing that slowed us down was all of the back and forth communication about bugs and fixes that needed to happen before we can ship. In the in-person world, what we would do is we would just pick up our laptops and we'd go round the office and we'd try to find the engineer we're pairing with and we'd hand them our laptops.

We'd say, "Here, it's in a perfect state. You can reproduce the bug on my browser and inspect it." But when the world went remote, there was no way to do that anymore and so we realized this was something that needed to be dealt.

Eden: And so for the sort of bugs that are reported by Jam users, is it more focused on functional bugs and regressions? Or is it more like usability feedback? Or is it both?

Dani: It's all of the above. Even to the design tweaks and copy changes. But the real thing that fuels us the most are these hard to reproduce bugs that are so painful for customers, and so painful for the engineers and internal teams. We're really excited about working to make those catchable, reproducible and fixable in a much more streamlined way.

The way it works is, in the real world when you want to fix something, you would say something like, "Oh, here, let me show you." So Jam is let me show you for software bugs. You install Jam on your browser and when you see an issue you can just click on Jam, and it will capture the last 30 seconds, instant replay of everything in the browser, from what happened on the screen to what happened in the console logs and the network requests.

What was the network speed of the device at the time, all the specs of the browser and the operating system and the device, and package that automatically into one link that you can just send off to a ticket or paste into a ticket, paste to an engineer. So the engineer just opens the link and they have a perfect playback of the bug and can see what may have gone wrong.

Eden: I see. I guess it's interesting because I feel like in this day and age, there's so many different test automation tools, and Selenium and Cypress and I had presumed, at least being on this side of the house, on the more mobile side, that everyone always tells me that web QA or web testing is already a solved problem or that if there is a bug, that you could just pull it back with some A/B testing or feature flag.

So it's surprising to me, or maybe not that surprising, to see that actually there is this need for a tool to help make bug reporting easier. It makes total sense. I remember when I checked out your website and I just saw the little product tour and overview. It's fantastic. But I'm just curious, do you see any patterns in the kinds of bugs that your users are reporting?

Are there more bugs reported by companies that have less automation tooling in place and are doing more manual testing? Or certain industries or sectors? What makes a good Jam user?

Spotting Bug Patterns

Dani: Automated testing is awesome, but at the end of the day if it's a human that's going to be using your software, then a human should be testing it too. Oftentimes we'll see things that pass an automated test that, to a human, feel very buggy and those need to be captured, reported to engineers. Often when they see the playback they go, "Oh yeah, that does not feel right." The other thing that most people don't realize about automated tests is just how much hard work it takes from an engineer team to maintain them.

Automated testing has always been the Holy Grail, the dream, right? Wouldn't it be great if every time you submit a PR, then every test runs? But the truth is just that tests break, the infrastructure that runs the tests break, and so it just takes a lot of maintenance effort. I was recently talking to an engineer at one of the FAANG companies that really is able to rely 80% on automated tests, and what he said is they actually have an on call rotation schedule just for those automated tests so that they can run.

And so it's a really heavy lift that most teams can't afford to make. I think the other thing is that in a software world, the software is the storefront, it's the back office, it is the whole business, and so it all meets together in the website, in the web app. And so you can't afford to have a loss in quality, and so you really want a person to look at things and go through all the details.

Something that we see across the Jam user base is that what makes for a good Jam user is someone who's product quality directly correlates to revenue and they just can't let that drop. One last thing to add is that often we think about the testing process being something that happens between QA and engineering, maybe product management.

But what we see in our user base is that with such an important emphasis on quality-- In 2023 quality is really becoming this full team effort and a lot of companies are requiring and asking for the whole company to be involved in dog fooding the product, testing it and giving feedback when things are not working or when they feel unexpected.

We now have a couple of customers where they require the Jam Chrome extension to be installed in every single company laptop because they want to make sure that if anyone sees anything that happens to go wrong, then it makes its way back to engineering. Because quality really is a whole company effort.

Eden: That's a fantastic thing to know, that you guys are really making it easier for the ownership and responsibility of product quality to expand beyond just engineers, beyond PMs. It can be anyone in the company, anyone is capable of reporting that. So I guess when you describe a Jam user, there's the person that is installing the Chrome extension, filing the bug.

But what are the patterns that you're seeing across the folks that are triaging, interpreting the bug reports? I'm assuming there's a different portal or a different user experience on the other side once all the Jam reports come in, right?

Dani: Yes. We want to make it super easy, two clicks for anyone to report bugs to engineers, and on the engineering side we want to make sure that all the technical details are there. It's a scientific report, it's as clear as possible so they can just go and run with it. What's really exciting about that is in today's world, the whole world runs on software.

Even the brick and mortar is all software ops underneath. A hospital is software, rocket ship has a lot of software, AI is software, and so progress today moves at the speed of software and so the more team members you can bring into the software process to communicate effectively with engineers, the better results you're going to get and the faster the future is going to arrive for everyone.

So that's really exciting, and this issue is engineers speak this very technical, very particular language that most of the company does not have training to be able to speak, and just to be able to bridge that gap unlocks a lot of conversations and communication that otherwise wouldn't happen. What we hear often from our users is they appreciate having more of a seat at the table now that they're able to communicate what they're experiencing, what they're seeing to engineers.

They're appreciating this more flowing teamwork that comes from not having this frustration of not quite understanding each other. And so that's super exciting for us.

Eden: Yeah. It's interesting to know that you guys are making it easier across the board for this communication to happen. One of the things that we've sometimes come across is that at least with some of the customers that use Mobot, they can sometimes get overwhelmed by the number of bugs that are being reported. What are the ways or methods or patterns that you've recommended for Jam users to make it easier for them to cut through the noise and find the bugs that really matter?

Using AI to Find the Bugs That Matter

Dani: This is super interesting. We hear lots of different takes on this from our user base. Something we hear often is a lot of people outside of the engineering team are nervous to report bugs because they don't want to bother the engineers, they kind of feel bad. But then we'll hear from the engineers on the same team that they actually really welcome the feedback and they want to hear, and some product managers are like, "No, give me the fire hose. More is better."

The other thing that's quite different here is one of the differences between automated and human is that a human can do the signal to noise in a really nice way.

When it's a human and they're working with an engineer or a set of engineers, there's this constant feedback loop that's going through the QA. They're constantly talking about, "Hey, did you like the format of this bug report? Was this helpful for you? Do you want more bugs like this?" So over time, that becomes a really strong partnership and they're able to cater to the engineer's needs. So we find that lasts for Jam, that I would expect for a robot powered tool like Mobot.

Eden: I see. I also saw recently that your company announced a new feature using AI. Would you be able to tell us more about that?

Dani: We're so excited. Most people realize that software engineering has changed so much in the last 30 years, but what most people don't realize is how little the bug reporting process has changed. And so that's what we're doing here at Jam, right? Is bringing the bug reporting process from this archaic, 90s, manual process into the future, which is similar to what you're doing at Mobot.

So we're excited to use whatever technology is available to support propelling the bug reporting process forward. When we saw what's possible with AI, especially with these developer tools like Copilot and just seeing the examples of ChatGPT writing code, we were just so excited to bring it in. And so we just launched JamGPT, it is an AI debugging assistant for engineers receiving bugs.

Oftentimes when an engineer sees a bug, they need a little bit of help either pinpointing where to look for the issue if they're more junior, or it just starts now as a Stack Overflow search. They now need to search, have multiple tabs open, looking for what does the error mean and how do I resolve it. And so JamGPT just shortens that whole cycle.

It has all of the context of the bug report, all of the logs that have happened, all the context of the web app and suggests for the engineer about where to start looking, what might be the problem and how to resolve it. We made it a chat so that it's an open conversation. The engineer can say, "Oh, that's interesting. Can you tell me more about that? Or can you explain that to me in a different way?

Or I now have this new information to include, what do you think about that?" Or they can even paste in code and JamGPT having all the context of the bug plus now this code can just write a fix for it, and even a PR description when they submit the fix.

Eden: That's amazing. I think that's definitely one of the pain points that I've observed is that sometimes... Especially going back to the earlier point of there's a barrage of different bug reports coming in, and you want to cut through the noise, make sense of it, and something like JamGPT, it sounds like it can really just help you pinpoint and hone in on the thing and then allow you to move through other Jam reports as quickly as possible. That sounds fantastic.

Dani: It's been a really interesting process because we're all learning as an industry how to leverage this new technology to accelerate software development, because software development is building the future and so it's really important to make it as fast and as productive as possible. We've been talking to tons of developers who are using Copilot and all these AI tools out there about how they're finding things helpful, how they're not finding things helpful, where people are seeing the future.

A couple of things are standing out to us. One is with AI you don't need to search. We've all experienced how we used to go to Google and now often we just open ChatGPT because we just need a concise answer and it's able to sift through all the noise and give us one thing. So that's a lot of software development is searching. There are all these memes of, "Being a software developer is just searching Stack Overflow," or about how many tabs in Stack Overflow you might have open.

And so with AI you don't need to feed the context into the knowledge base of the web, the knowledge base of the web can get the context from you and come to you and give you the one answer you need. So that's quite interesting and I think it has really big implications. We're really excited about leaning more towards that.

The other thing that we're hearing is in 20 years, maybe AI is writing the software, but until then it's really a human plus AI partnership that is really the 10X engineer. AI is really good at these discrete, concise, limited scope questions. It's really good within a file. But a code base? It's much more complicated, there are multiple services, there are multiple repos, they're very large.

There are all these implications around business logic and user experience that the AI can't quite do yet, and so it's when the engineer can leverage what's possible with AI that you, as the human thoughtfulness and the design and the experience and knowledge in architecture to bring in a really nice way, is where we're seeing results.

So those are the design principles behind JamGPT. We're thinking about how do we eliminate search, and how do we partner the AI with the engineers.

Eden: I think that's a great way to articulate and a great way to think about it. Especially nowadays, some of the conversations I'm having about AI is people are just worried about how is AI or automation going to take all of our jobs or, "I'm not going to have anything to do anymore." And I've honestly, genuinely just found, kind of like you, that that's actually not going to happen.

You really need to be so specific, so discrete with the AI, with these very particular challenges or prompt engineering that you have to do. I think the opportunities that open up for humans and the role of product people, the role of engineers, the role of QA is going to certainly change, but there's not in a way that where that's not needed anymore.

I'm curious what your experience has been like. We were chatting about this before we started recording and it sounds like you and I have had similar conversations with customers and users.

The Evolving Role of QA

Dani: I'm actually quite excited about how the role of QA gets to change as tools make QA more productive because the QA mindset is so valuable in a company and in the engineering organization, where quality is front and center. QA thinks about quality in a different way, they think about testing in a different way, and so the more time you can free up in your QA team to get involved earlier in the process, the better results companies can have.

Something that we've heard over and over again is that with the rise of remote, suddenly QA that was often done outside the office is inside the Slack. So it's on the same wavelength as the engineering team, and suddenly what that means is that QA can be a real partner to engineers.

Both as like there can be a collaborative feedback loop that helps the QA provide exactly what each engineers needs, boosting the engineering team's productivity even more. But also that the QA team can be involved from the product manager writing the spec to the testing phase. They can go out and think through what are some edge cases that engineers may want to think about ahead of time, versus at the end of the cycle. This helps companies ship faster because instead of having at the end of the cycle these tons of slow communication cycles, iteration loops, a lot of planning and development can happen out front to prevent that.

So that's quite amazing, but if QA is so busy, going through all these regression tests, trying to just make sure the engineers are on block and that features can ship, they don't get to do all that work. So I think that the role of QA will become even more important, even more ingrained in the engineering cycle as these new AI powered tools allow them to do more.

Companies with these AI tools want to ship more, not less. Just imagine you hear that your competitor is leveraging AI to ship twice as many features. You're not going to be like, "Great. Well, we're going to make cuts." You're going to be like, "Well, we're going to do that too and we're going to see if we can leverage AI to ship three times as many features."

And so I think these roles will change a lot. I think there's going to be a lot of exciting new ways that these roles will become even more architect focused, even more strategy focused, even more product focused in a way that I think will actually be quite creative and enjoyable.

Eden: Yeah. I think especially now more than ever, people are really starting to open up and realize just how much more strategic we could be about testing. If you had time to actually tactically think about when to test, should it be pre-production, post-production, the way that we think about testing and monitoring and integrating all of these developer tools together, that is really going to change because we're not just caught up in the mundane, manual testing anymore. I think that's going to be something that tools like JamGPT and Mobot and just the normal Jam platform are really going to unlock, which is super exciting.

Dani: There's been this movement in QA for the last five years about shift left, which is if you imagine the product development process, the software engineering process on a timeline where QA was all the way to the right, then it gets to shift left, earlier in the process. It's been this very talked about subject, it's a very exciting subject, how can we bring into it earlier and what are all the impactful ways that that contributes to a more productive software engineering process.

But until now it's been really hard to achieve for teams because QA is quite busy. They have a ton to do and so actually these tools that free up a little bit of their time so they can be more strategic partners in the beginning is super exciting for engineering teams.

Eden: Yeah. I'm actually curious since I know you've been running Jam now for the last three years, and you also worked at Cloudflare, and a couple of other roles before that. How has your perspective on QA or testing or product development evolved from when you originally got the idea to start Jam to now? Do you feel like your philosophy or your perspective has changed? Or has any customer feedback or user feedback surprised you?

Changing Philosophies on QA

Dani: One thing that's been very, very interesting is as a startup, we've been learning a lot about what quality means for startups like us. I think that there is bad startup advice out there, which is that as a startup, you can afford to have bugs. You should ship as fast and messy as possible because that's the only way to test your product. I think that this is outdated advice that no longer works, and I believe based on our experience that the opposite is now true, that the only way you can test your product is by putting out a high quality, bug free product out there.

The reason why, is 10 years ago when the advice was ship fast, ship messy, move fast, break things, the world had not yet changed in the way it has today in respect to remote, mobile, SaaS. Today our lives are running on software. When you show up at work if you work at a remote company, you're showing up in various software tools and if those tools don't work, simply, it's unworkable.

And so in today's world, quality really is front and center and there are no exceptions, even for small startups. We see that actually in our user base, so I think that five years ago, ten years ago, it was very uncommon for a startup to employ QA. But what we see now in our users is that even three person, four person, five person startups also have someone working in QA, and that's just because the bar is higher and it's so important to have a high quality product.

There's also one more thing here for startups, which is when you are just starting out, the most important, first thing for you to figure out is what exactly is it that you're going to build that's going to solve you user's problem? And it's actually quite tricky because you get a lot of input and it's hard to find the clarity in that. The more bugs you have, the harder it is to find that clarity because when users leave, you don't know if it's because they couldn't use the product or if they didn't want to use the product.

Often because they don't want to hurt your feelings, they often will just tell you like, "Oh, it didn't work."Or, "There was this bug." And so really you need to ship a bug-free product in order to gain clarity just for yourself. One thing that's really changed for me in my perspective is about the importance of quality, not just for the big companies out there with millions of users, but for the startups just starting out who want to start fast and start on the right foot.

Eden: I actually really agree with that. I think quality has become a differentiator because right now everyone... You can write a web app with ChatGPT or whatever, that's what I hear. There's all these YouTube tutorials that are popping up so anyone can write code and no-code platform, all that sort of stuff. So yeah, what differentiates your platform from someone else's platform is that someone can actually use your product in the way that you and they intend.

So that makes a lot of sense, that that startup advice that people get needs to get challenged. I do think that there's all these... As founders and CEOs, we get a lot of this advice from investors and fellow founders and accelerator programs that we go through and all of that, where when you're so lean and the resources are so tight and you just want to focus so much on iterating.

But yeah, I think the way that you put it really made a lot of sense to me as well because I've definitely heard bad advice when I was starting Mobot a few years ago. I do agree, the bar has been raised. You just can't get away with it anymore because if you don't build a high quality version of that product, someone else will.

Dani: Yes. Move fast and break things just doesn't work anymore in 2023.

Eden: Yeah. Moving to the part of our conversation about you being a founder. In your role as a CEO, I'm curious what your day to day is like these days? Is it very product focused? Is it more business focused? Do you spend a lot of time talking to customers? Just curious what it's like.

Dani: I love our user base. One thing that you don't realize when you first start your company is by what you choose to work on, you are choosing who you're going to be spending all your time with for the next decade and we got so lucky that we chose people who are building products and are trying to do so faster. I am so lucky to get to spend all this time talking with our users, with our customers, because they're all trying to push the world forward in some way and Jam is one thing that helps them do that faster.

It's really exciting to talk with them about what impacts they're making and how we can help them do that better and faster. As a founder in this moment of Jam, it's so exciting. It's such a moment in the company. We launched Jam exactly a year ago, 13 months, and over that time, 25,000 people started using the product from companies that we really admire and love.

Product companies like Peloton and Ramp, T-Mobile and Staples. So now a year in, we're at this moment where the company is growing. When you start a startup, people tell you you're not building a product, you're building a company. But the first phase is really very product focused and suddenly in this moment, it's growing on its own through word of mouth.

We get to lift up our heads and say, like, "Okay. Well, then how do we build out all the rest of the company stuff?" And so my focus up until now has been extremely product focused, but I'm excited to now do the other bits of the company and focus people on building new functions. We just hired our first marketing team to help support building a community around Jam.

Our users are so chatty with us, they love sharing with us how they're using it and what they love to see, and we want them to meet each other, all these amazing product builders all over the world.

Eden: Yeah. One thing that I thought is really cool about the platform you guys have built is you can really leverage a lot of best practices with product led growth. You don't need a whole team to sign up for Jam and start using it, you could have one person from a company sign up, share a report with Jam and then... I'm curious if you could tell me more about that strategy of how do you embed Jam into the developer workflow? Starting from that one person who then sends a bug report, how does that process work?

Embedding Jam into the Developer Workflow

Dani: It's been such an interesting learning experience, and an amazing discovery for us as a team. When we first sat down, we said, "Okay. How are we going to grow this?" And we would read all these blog posts and listen to these podcasts like How Notion Got Its First 10,000 Users, and How Figma Got Its First 10,000 Users.

We were trying to figure out what's the marketing hack, what's the channel, where do we run our ads? Sign us up for the Superbowl. But actually what's really happened is just by using the product, people naturally share the product because when a person captures a bug with Jam they share it with an engineer, and their teammates will see the Jam either the link in the ticket or they'll see the ticket created by Jam, and engineers will all see it.

They'll ask other people in the company to use it because they like the format. So really, for us it's just been focusing on the product and making sure that it's really valuable, so that it gets people using it and they retain and they're using it for a long time and they use it more and more.

That's actually the way that Jam spreads. Up until this point it's just been word of mouth. Marketing doesn't grow Jam, product grows Jam, and so the best time that we can spend is just on improving the product.

What I love so much about that is it's so incentive aligned. It's like if we provide value to our users, then they help us grow so that we can do even more of that, which I really, really love. What's cool about this moment is now with some validation that, "Oh, okay. This is working and it's growing on its own." Then we can say, "Well, great. How do we accelerate and get more people to learn about a new way to do bug reporting, to speed it up in their company so that we can do that even more?"

Eden: Yeah. What's next for you guys on your product roadmap in the next one to two years? What are you focused on building out? Is it more JamGPT or is it something else?

The Future of Debugging

Dani: I think the future of debugging is in AI. I think that AI is going to transform a ton of dev tools. A lot of dev tools have a lot of signal to noise issues. Anywhere someone is searching through logs, AI is really good at summarizing. Anywhere that the scope of problem is single file, like config file, infrastructure as code, Kubernetes YAML files.

All of that, AI is going to be so good at, unit tests, and so that's going to be really amazing. We're going to be spending a lot of time there. But really, truly, we want to make it a lot more productive, a lot less frustrating to build software and we want to make it a lot more productive and a lot less frustrating to use software.

Anywhere that people are building product and they need to say, "Let me show you what's happening to an engineer,"that's where we need to be and we're so excited to go there.

Eden: I'm really looking forward to see what features you guys ship. I think it's a fantastic tool that you've built to make it easier to have users connect with engineers for web apps. I'm also curious, are you guys thinking of potentially expanding to mobile or other interfaces and other integrations as well?

Dani: Anywhere someone is reporting bugs, that's where we need to be. I'm so excited to jam with you at some point about mobile, what you're learning, what you're seeing. That's our number one, most requested feature, is some way to Jam on native mobile apps. That'll be so interesting for us to figure out how we can do that.

Eden: Yeah. I think there's a lot of opportunity which I think about the problem in a few different ways. There's bug discovery, which is something that Mobot spends a lot of our time on, where we have the robot running through different scenarios to try and find the bug. But yes, one of the things that afterwards is once you've found the bug, how do you make sure that the right information is provided and especially with some of the ways that Apple and Google-- Well, it's actually easier on Android than it is on iOS, to access the network traces, the iOS device console log. Some of that stuff, some of that data which you really do need to actually debug, it can be a little bit hard to retrieve sometimes. I would love to, when you guys are ready to explore that, we should definitely talk.

Dani: Amazing.

Eden: Yeah. Thank you so much for joining me on the podcast, Dani. This was fantastic and I'd love to catch up again in a few months and see how much the product has evolved. I'm very curious, especially to check in about AI and how JamGPT evolves as well.

Dani: Amazing. Thank you so much for having me on, it was fun to jam with you.