Ep. #16, Language Translation with Spence Green of Lilt

Grant Miller: All right, Spence. Thanks so much for joining.

Spence Green: Grant, thanks so much.

Grant: Cool. Let's just jump right in, tell us a little bit about your background.

Spence: Sure. I've been interested in computers since I was a little kid, so I get to do exactly what I've always wanted to do my whole life every day, which is great.

I majored in computer science in college and then after college I was really interested in military systems, specifically on planes, so I went to work for a defense contractor working on large military systems.

While I was doing that I was part of that time overseas and I got interested in languages and information access, and it was right around that time in 2006 that Google Translate came out.

I bought a book which was full of statistics which I didn't understand, and I realized that I really wanted to work on this technology but I lacked the mathematical skills and computational skills to work on it, so I left for grad school in 2008 to start doing machine translation research.

Grant: Interesting. That was inspired because you thought that Google Translate was really interesting, but you weren't really grokking all of the different math behind it?

Spence: I didn't understand until grad school that computer science and software engineering are not the same thing, especially in machine learning.

It's just really math and statistics, and my mathematical maturity was pretty low, even though I was a good programmer.

I had to go to grad school to develop those tools to be able to work on these systems, which now people talk about "Machine learning," but in those days there were no frameworks.

If you wanted to write an optimizer you wrote it from scratch.

First optimizer that I wrote for our machine translation system was about 6,000 lines of code, and you can do that in one line of tensor flow these days.

There's been tremendous progress in making this technology more accessible, but none of that existed 10-12 years ago.

Grant: So you went to Stanford, started getting your Masters and eventually you got your PhD there, right?

Spence: That's correct. I started in the Masters program because I really had no research background, and to get a get into a good PhD program you have to read paper, so I built my research credentials so that I could go on to a PhD.

It was right around that time that I was working and I had to find a research direction, and that's when I met my now co-founder John, and picked up on a thread of research that's been going on since the late 60s that nobody had ever really gotten to work.

That became my dissertation, and then a lot of the work that we did together, and now the fundamental technology in the company.

Grant: Cool. Pretty strong academic routes here for Lilt then.

Just talk quickly about Lilt, the company you've started.

Spence: Sure. We provide a translation of text to institutions, where "Institutions" means large enterprises and governments.

Grant: Cool. OK, so you're at university getting your PhD and you find this older body of work that --

Why had the work stopped on that?

Spence: It never really stopped, but it was at the intersection of natural language processing, which is my field, and human computer interaction.

It's interactive machine learning, and now people call it "Human in the loop" learning.

It sits at the divide between these two fields, and there's not really many people working there.

You really need a specialized machine translation system to support and work with a human, and then you need a really specific type of interface for the human to work in.

There really hadn't been somebody who had been spending equal amounts of attention on both the back end technology and the user interactions, so I got a co-advisor from HCI and then I co-advised for the second half of my PhD.

We made contributions to both developing interfaces for this task and developing the back end technology.

Grant: Cool. From the enterprise use case you're basically making it easier to translate all of their internal documents and marketing, what's the--? How are folks using it?

Spence: That's exactly right. So, John-- Our initial interest was in books when we started working together, "Why don't more books get translated?"

Recall, we're approaching this from the perspective of MT researchers working on Google Translate, which is like "Just put some text in a website and it generates a translation. What's so hard about that?"

We started to understand how companies do translation, and we were amazed to discover that in those days and really even today, very little technology is used.

The companies that serve enterprises are services boutiques. They operate very much like law firms, they hire a bunch of professional freelance translators who they give work to and do project management with, and then they give that work, the text, back to the enterprise.

It's human labor so they try to mark it up as much as they possibly can, which is why stuff doesn't get translated.

It's like working with a law firm. They're just trying to mark up as much as they possibly can, and we're approaching it as technology people, which of course we want to drive it down as much as we possibly can.

So there was just an impedance mismatch between what we wanted to see in the world and what this industry was structurally set up to do.

Grant: OK. So then, was that the "OK, we should start a company around this?"

Spence: It was more pragmatic than that, I suppose.

My advisor, who I owe a great deal to and was very influential in my life and continues to be, but he a couple of years into grad school he said to me one day, "It seems like your work is pretty applied. You ought to learn about companies."

I started spending some time at a venture capital firm just listening to pitches and trying to studying this market.

The quid pro quo is these investors would help me understand this market that I was trying to understand and I would do due diligence for them, because in those days--

This was like 2013 and the first couple of companies pitching machine learning was just really bad, so I'd just sit there and say "No, that doesn't work."

And then John and I faced a choice, which was -- OK.

We got the technology to work and we knew it was going to happen, it was inevitable, and so it was like we could stay and research, keep publishing papers and create an open source project.

We could try to go to a big company and advocate for building something like this, or we could start a company, and John more so than me felt like the highest impact route and the way to get it into the hands of users fastest was to start a company.

Even though we had really no qualifications or experience to do that, with the exception of I'd been hanging around a venture capital firm trying to understand how companies are started.

Grant: Sure. But I'm pretty sure that describes most entrepreneurs.

We don't have any qualifications to start companies, only the second time or third time do you feel like you actually know what you're doing. First time you just jump in.

Spence: Yeah, I think that's true, although I would wonder if you looked at the successful outcomes in enterprise.

My hypothesis would be it's usually second-time founders.

Grant: I think that's probably less true on the consumer side and much more true on the enterprise side.

Spence: I think that's true.

Grant: It's where execution risk becomes-- Like, you can take away execution risk.

Spence: Yeah, that's right. On the consumer side, I was listening to an interview with Justin Tan recently, who's recently started this technology-enabled law firm.

Grant: Right. Not too far off what you were talking about.

Spence: There's a whole class of businesses now that are technology-enabled services.

It's happening in software testing, it's happening in localization, it's happening in legal, it's happening in bookkeeping. In the consumer side you have market risk, so lean startup principles apply.

Figure out if there's a market there as quickly as possible, and it's like throwing spaghetti at a wall.

In enterprise we know there is a market for law firms, there's no market risk to sort out there.

There's a lot of execution risk to be able to do what Cooley or Gunderson does.

That requires people who are expensive, expertise that is specialized, and you need to look like a business and present as a business long before you actually are a business.

Grant: Yeah. It's interesting because I feel like it sounds like Lilt was really--

The only difference is it's really core technology that you'd been building and focused on, and you had this truly deep subject matter expertise around machine learning for translation in a way that maybe only Google and Microsoft did.

Spence: That's about it, yeah. There are a couple other companies, but really in terms of building large scale production systems Google and Microsoft are-- And IBM.

I will include IBM in that class as well, but experience building these large scale production systems, that's true.

Grant: OK. So then you decided to start a company, and your co-founder-- Did you guys jump together? How did that work? How did you guys split it up?

Spence: John was leaving Google to go to Berkeley to join the faculty, and the third person involved in early days was Franz Och who started the group at Google and is a pioneer in our field.

He was leaving Google as well, and Franz had worked on this idea in the late 90s when he was in grad school and then had gone off in a different direction.

He was very encouraging in the early days when I started, when I picked up this line of work.

He advised us a lot and helped us a lot in the early days of thinking about the business and thinking about the product, and so on.

Grant: Cool. So what did you do? You went to market with this translation technology, and how did you start to get in front of people? What was the plan?

Spence: Our first hypothesis was just completely wrong, as were our next four hypotheses. It was the business model that actually worked.

Our industry is this outsourced, multi-step supply chain, and our fundamental insight was that if you could use technology to enable a translator to do more work per unit time, then you could compress the cost of translation.

We also had results from grad school days that you could use technology to increase quality as well, so you can make them more effective and you can make them faster and you can monetize that productivity to reduce cost.

So we thought we would sell a tool for $20 a month to translators and they'll buy it. For them, it's a margin gain.

They can do more work per unit time, charge their customers whatever they want, and keep more of it themselves.

Grant: Are translators generally independent contractors?

Spence: Yeah Over 80% of the workforce is independent contractor.

Grant: OK. So you have all these independent contractors, they're doing this part time, full time, combination?

Spence: Some of them do it as a second job, a lot of them do it as the first job but it's episodic in nature so they alternate between looking for work and doing work.

Grant: But the idea is that, "OK. We can make them more effective and they're going to deliver better results, this is going to help them make more money, so they'll get it."

Spence: Exactly.

Completely rational to us, and it was just completely wrong.

Grant: What was wrong about it?

Spence: In this supply chain every link in the supply chain dictates technology and workflow choices to every link further down, so these translators, they work for a services boutique and the services boutique tells them what tool to use.

The services boutiques tend to work for bigger services boutiques, and those bigger ones tell the smaller ones what to do.

The bigger services boutiques work directly for customers, and customers oftentimes tell the big services boutique what to do.

It took us longer than it should have to figure this out, but it's very difficult for any of these services boutiques to standardize on one production workflow.

Therefore when they're looking at a new tool it's always like the 15th tool that they might have in their inventory and they can't use it for all their customers.

Therefore they're not willing to spend a lot of money so the deal size is very small, and this technology that we're building is very expensive so the economics did not work.

Grant: Right. Because you mentioned before this you guys maintain a research team. Like, you have--

Spence: Yeah. We have six or seven people in the company with PhDs.

Grant: That are actively trying to improve your-- I don't know what you call it, your "Underlying engine" or technology or however you describe it.

Spence: That's correct.

Grant: And that's expensive.

Spence: It's not cheap. They don't work for free.

Grant: Yes, exactly. They are very altruistic, but they don't work for free. So how did you discover it was wrong?

You were trying to bring it to customers and they just wouldn't pay enough for it? Or, what was the--?

Spence: It was just one false thought after another. "OK, here's our next hypothesis. We tweaked the product a little bit, tweaked the positioning, and we're trying to run a sales cycle."

We'd win five customers or something and we were like, "Yes, this is it."

And then it wouldn't be able to get any more customer, so the breakthrough came about a year and a half ago when we realized that we couldn't partner with anybody in the industry because their whole industry business model was structurally aligned against what we're trying to do.

So what we decided to do was to start a services business ourselves and vertically integrate.

Now we go to a customer and we provide the human translator and provide the software as well.

When you internalize the complexity of the whole supply chain, then you can optimize across it and you can have strict SLAs for speed, quality, and on-time delivery. You can optimize for cost, it just affords a lot of opportunities that you can't do when you're depending on vendors.

Grant: Right. Because when you're working with that vendor supply chain, you're just a part of it. Basically, even though you're making it more efficient, anyone can take the efficiency out of it.

Spence: That's right. You're optimizing one link, and it may be that link is not on the critical path.

Grant: Right.

Spence: So you've got to own the whole thing and then just systematically knock out the critical path.

Grant: OK. That sounds like a pretty big undertaking. Were you able to--? How were you able to expand?

Do you just work with a customer where you were the entire supply chain, and keep going up the stack? How did you get to that?

Spence: It was less elegant than that. We made the decision, it must have been about two years ago now that we were going to--

One benefit of climbing the supply chain was that we understood how the supply chain worked.

That was a painful experience, but it did give us the operational knowledge to set up a services unit ourselves.

Grant: When you say "Services," this isn't professional services, this is actual people who are doing translation, right?

Spence: Yeah. It's like professional services where we source, qualify, and manage these independent contractors and we train them for each customer account that we work on.

It's not professional services in the sense that you're used to, which is like setting up on-prem software, it's the "Human in the loop."

Grant: Right. They're actually doing the translation. They're part of the product.

Spence: They're part of the-- We think of it as a production process. Our whole company is a factory.

Words come in one side, they go through a bunch of production stages, and words come out the other side with certificates of quality.

There's an on time delivery SLA, and stuff like that. The human translators are essential factors in the production process.

Grant: OK. When I think about internationalization of products, that's my familiarity with a lot of translation.

When I was at an enterprise software company we had to internationalize and basically make our product work with a bunch of different languages.

Is that part of this? Is that part of what Lilt is offering? The ability to "Put in these strings and then we'll substitute out all the different languages for you?"

Spence: That's exactly right. Product interfaces, support pages, marketing websites, product documentation, media articles. Anything that's written down.

Grant: How far into web translation do you get? Do you have JavaScript that goes into the page that's then subbing things out?

Or how do you--? Like, managing the database on the back end. How far--?

Spence: That's the enterprise integration point with our product, which is we usually connect directly to the CMS or the repo where the linguistic content is stored.

By the time it's rendered as a web page it's come out of a CMS and been confined in various ways and rendered in a view.

We typically work with the raw linguistic content, which is stored in some repository.

Grant: OK. So either a CMS, or a get maybe.

Spence: A source, yeah. For user interface, there's a resources file or something that's in a get repo, so we connect to that.

Grant: Cool. That's super helpful. OK, now you made this realization you needed to vertical ize and to take the entire supply chain.

What next? Was it just instantly everybody understood and they were able to start using it?

Or did you have to start to change your product, add more features? What was the--?

Spence: Yeah. As you probably know, painfully in the enterprise it's about-- For the initial sale, it's about 95% sales and marketing and 5 % product.

I think for the renewal then the up sell product starts to matter, but a complex multi-stage enterprise sales motion is-- There's so many ways you can fail and lose in that before you even get to servicing the customers. That's the part you've got to learn and get right.

Then suddenly you go from a little web page that we had with a credit card sign up hooked up to Stripe to now a six month sales cycle, or whatever it was at the time.

It's harder to run experiments, like "OK. I'm going to go to enterprise now, and I'm not going to know whether this works for months."

Grant: But you knew that it was - - I'm guessing you were interviewing some customers and talking to a few people.

Spence: We were interviewing some customers, it's true.

Again, the benefit of going into a mature market is that you know how they buy things right now and you know what their budgets are.

They're spending a truckload of money on this function, it's just figuring out how to differentiate and in our particular industry there's all sorts of non-financial dynamics like relationships.

I've worked with this person for 10 years and they send me a Christmas card and we're Facebook friends, and for an engineer that was very difficult for me to deal with.

Like, I want to show up with a results table and prove to you that this is better and you're going to save millions of dollars, and you're telling me you're not interested. I don't get it.

Grant: Yeah. That's the human side of enterprise software, which is why sales people exist.

Spence: That's exactly right.

Grant: Because they help figure the human element out and navigate those relationships.

Spence: Yeah. We talk about this a lot with our sales team, but they manage complexity on behalf of the customer, or something.

They manage the complexity of solutions in the world and match that to what the customer wants.

Grant: Sure.

Spence: Especially in a consultative sale, which ours is.

Grant: What's your average order size? Is it in the hundreds of thousands of dollars per year, or is it?

Spence: Yeah.

Grant: Great. So, pretty high ticket item, you're coming in here.

But to your point, you're actually displacing some existing budget maybe in some incumbents.

You're not trying to convince these enterprises that they need to do translation, they're already doing it.

Spence: They're already doing it. Every business of a certain size does it.

Translation is bilingual writing, and I think every business with the internet-- We think it's an insult that you don't provide a native language customer experience to all of your customers.

Eventually we would like to make it possible for everybody to do that.

It's too complicated right now, and I'll give you an example, we do not provide a native language customer experience to all of our customers because we didn't build our software the right way and we didn't build our website the right way.

So we're in the process of re-architecting everything, because there's an engineering dependency too.

Hard coding strings into things and you've got to redesign all your interfaces and pull all the strings out. We were clueless about it too.

Grant: Yeah. When you make that-- I've done it twice. It's a real feat.

There's a lot of lift, and if you didn't set it up that way to start, which most people don't--

Spence: Then you're in a hurry, you don't know if you're going to fail, you can get traction in English.

That's completely rational decision-making, but then at some point you have to retrench and make it a priority.

Grant: The hardest part for me was actually getting my mind around RTL, right to left languages.

We just still-- When I was trying to understand how to build in translation for languages that you would read the other direction, I was like "How do you even look at a page the same way?" It flipped me.

Spence: Yeah, that's right.

Grant: And there's some other interesting pieces just around, particularly for web design, where words and things become integrated into specific sizes and different languages can expand outwards.

There's some really interesting challenges in there. It's not just about web pages, it's about all the content that people have.

Spence: Any text, any strings, anywhere. We don't do speech, but that any text anywhere.

Grant: Which the internet has a lot of.

Spence: There's a lot of text on the internet, it's true.

Grant: You start to vertically integrate, you're getting some success.

Was it a specific first big enterprise deal that made you realize this was it?

Spence: Yeah, we won a couple deals. We won the first three and then that was a false dawn, but we kept going because there was nowhere else to go.

Grant: Because you tried different angles on this?

Spence: We'd explored the entire supply chain and vertical integration was the last resort.

You can read my core competitive strategy, "Vertical integration is a rational strategy for consolidating a fragmented market."

I think you've got to think very hard before you're going to do it, because it doesn't work in all business. PC industry is a good example.

Vertically integrating a PC company, Apple tried it and they have 2% market share.

Turns out it's much better to have a bunch of component suppliers, so it doesn't work in every market, but I think in some markets it is the right strategy. It did turn out to be the right strategy with mobile phones.

Apple reaps enormous profits from vertically integrating the phone. But for us, we closed a couple of deals and we worked really hard to make those successful.

We were also running short on cash at that point, and we were always undercapitalized and understaffed because we always raise money for a B2C type go to market motion and then we raise money for an SMB go to market motion.

Then midway through that we decided we had to go to enterprise. We were always understaffed.

Grant: How big was the company when you made that transition, can you remember?

Spence: To enterprise?

Grant: Yeah.

Spence: 6? 5? It was rough, and I was doing the selling and I'm not probably in the top 10 million sales people worldwide.

It just took us a while to get a couple more deals to where there was enough proof there that we could raise enough money that we could hire the functions--

We had engineers doing CS, I'm doing sales, it was not good.

Now we have the functional areas that a business expects to work with, and a partner, but we just never had the money for that before.

Grant: But you were able to still close those deals and get enough proof points that you were able to raise your next round?

Spence: Yeah, that's right.

Grant: Is it just that your customers loved the solution, it was clearly better?

What's the thing that drove this as an obvious or as a good business?

Spence: Unit economics. The unit economics of the business are undeniable, and we had proof for that even though we didn't have a ton of customers to show for it.

The technology is inevitable. The technology has always been, since five years ago, it's been inevitable.

So you put those two things together and you can see what the future looks like even though there's a lot of hand-to-hand combat to get there.

The hand-to-hand combat is execution, which is the hard part.

Grant: Sure. OK, so just the unit economics of "How much does it cost you to correctly translate a document" versus "How much does it cost the rest of the system?"

Spence: Yeah. If I can use technology to make one person five times more productive, which we proved a long time ago, you can do something with that even if there's an enormous amount of resistance in the market.

Grant: Part of the challenge is you've taken four different approaches to the go to market ,and then you're finally trying to go to enterprise.

By that point you're pretty burnt out. You're like, "Investors are not seeing a lot of success."

Your employees are tired and you're just trying thing after thing. How did you push through that?

Spence: I think this is where having an academic background was enormously helpful, because we viewed it as just running experiments and most of your experiments fail.

The repeated failed experiments didn't bother me so much, I think what did bother me was the irrationality of it.

I still think it's irrational, financially it didn't make any sense. But you've got to realize, especially in the enterprise, that people are not--

They're oftentimes making decisions that aren't even aligned with their own business, they're aligned with their own career or what's going to make them do less work or what's going to get them some promotion or raise, or something like that.

You've got to figure all that out in each account, and that's not to criticize or anything.

I think anybody who's done enterprise sales knows that there are a bunch of interpersonal factors that you have to deal with. They're not purely financial.

Grant: Yeah, that's a really great point that there's a lot more under the hood. I think as engineers we often think it's like, "Clearly the best solution will win."

But you have to think about it from the perspective of each person who's impacted by it.

Spence: Yeah, and things start to matter like having in-person meetings. I'm flying to the East Coast next week for 20 hours to have one meeting.

If you'd told me a year and a half ago that was a good idea, I would have thought "It's bad for the environment, it's bad for your health and it's just a complete waste of time."

But it turns out in the enterprise that it really does matter.

Grant: Yeah. Some of the other things you probably had to build along the way, I'm guessing when you were trying to sell to the pro-sumer folks there probably wasn't even the idea of a team.

You probably couldn't manage and invite other people to do stuff together.

Spence: Yeah. We've been in this about year-long journey of implementing role based access control.

It seems simple in practice, but it touches every part of the system and it's just been a huge execution challenge for us.

But we've grafted on this sharing function, but as you say the basic things that you need in enterprise software like audit logging, access control--

Grant: Reporting.

Spence: Yes. Single sign on, all that stuff, we're having to build back into the product later on because it was never really designed or set up for that.

Grant: So even today, do you have a single sign on solution? Or, how do--?

Spence: It has Google OAuth.

Grant: OK, cool.

Spence: Yes, it's got that. That's really all that our customers want.

Grant: Yeah, that's definitely probably the fastest way to a single sign on solution.

At some point, you end up building out a SAML thing.

But Google has some SAML integrations so generally the only pushback you end up with is the folks who are like, "We're Microsoft. We use Microsoft everything."

Spence: Or you're trying to sell in China, where it's a big problem.

Grant: Right, even my trips to China trying to even use Google Maps was probably the biggest challenge. I was like, "Wait. That's how I get around. What do I do?"

Spence: Yeah.

Grant: Cool. So now that you're feeling this pool, you're seeing these requests and you know that "We have to get some of these features built into the product in order to continue to grow and get some of these enterprise deals."

One thing that we often hear is that enterprises are somewhat forgiving around some of these features as long as you commit to building them.

Spence: I think it becomes much more important to--

Even if, there's some companies that have a public-facing roadmap, which I don't think it really matters in the enterprise, but you definitely need to have guidance that this is going to be done.

The timeliness of doing it, our customers are not like "This has to be done next week."

But if you tell them it's going to be done in two months, you better get it done in two months.

I've found it's very difficult to get engineering teams-- It's like training an athlete or something.

Engineering teams around the idea of "There is a deadline and we have to hit it. If we're not going to hit it, we have to re-scope so that we do hit it."

I think that's peculiar to software engineering, because in the physical engineering disciplines "The rocket is going to take off at some time. You have to stop and prepare to launch the rocket. The bridge cars are going to go over it at a certain time, you just can't keep pushing it off. The building-- People are going to ride the elevators at some point and it has to be done." I think that it's much harder to get engineers conditioned around that idea that customer expectations have to be met.

Grant: Yeah, it's true. It's important to qualify this. You are the CEO, but you're also the engineering leader up until very recently.

Spence: Until last week.

Grant: Yes. You say this not from the perspective of the CEO top down, but from someone who's in the weeds building software with the team.

There's a discipline in an enterprise software engineering that I think it's just different.

You have to meet commitments, you have to think about stability, security is paramount.

Spence: Regressions and the tolerance for that is very low. It's like you say, you've got to think about security much earlier than you would maybe at a consumer company.

Although I think the wind's blowing in the world are changing that a little bit right now, but still you're going to go through an infosec process when you're selling to an enterprise customer and you've got to have documentation.

You've got to have some plan for GDPR and SOC-2, and if you're selling to the government for Fed ramp, and a timeline for all that. You just don't have to do that in consumer settings.

Grant: Yeah. In the consumer, pro-sumer world you're selling your software to people who maybe if you get really huge they start to be concerned about how you handle their data.

But if you're just going-- You and I probably think about it, you probably don't put a lot of data into other random tools that are hosted because you understand security now, but much of the consumer pro-sumer world isn't concerned about it.

Spence: That's right. I think that's probably a fair claim.

Grant: Yeah. Then one of the things we were talking before is you guys made a pretty big investment in Kubernetes early on.

You love it, you think it's been hugely helpful?

Spence: Yeah. The core feature of our product is predictive typing, and so what it looks like is when you're typing a message on your phone and it's predicting words and phrases for you.

Or now you see this in Gmail with that smart compose feature where you are typing and it's predicting words and phrases.

That's what ours does, only it's bilingual. I'm reading in Arabic, I'm typing in English, and it's translating every time I type a word it gives me a new translation.

You've got a budget of about 300 milliseconds to get a translation back, otherwise the system feels really sluggish and it's unusable.

It's been a multi-region system since the beginning and it's a bunch of different services, so it got to be like every time we were pushing out software you had to log into the region.

We had a bunch of instances running and you had to log in to each machine, pull the Docker image over and restart the thing, and I had an engineer and every time we did a release it would take half of his workday to update all the clusters around the world.

I guess it was in early 2017 maybe, we decided that we would move to Kubernetes.

Grant: That's right as that became not an insane decision, prior to that it was pretty early and it was still very questionable as to like if it was truly production-ready and "How much could it scale?"

Spence: John had worked with Borg and everything at Google and I'd used it a little bit when I was there.

I worked for John, that's how we met, I was his intern. Those ideas have been in Google for 100 hundred years or whatever, so we knew "OK. That's what we want to do."

But we ended up rewriting big parts of our system, and you end up writing these enormous helm charts to deploy this small micro service, and that's a little bit alarming when you have to start doing it.

But now we've got a really nice setup where all the regions around the world, you tag a release in GitHub and it goes off to--

We run in Google cloud, so it goes off to TCR where it builds the Docker image and then it does a stage in incremental roll out in all the regions around the world.

We can roll out specific micro services or we can roll out the whole system, like I was saying, doing a roll out.

In enterprise you've got to think hard about roll back, if you have a regression, your customer will call you real quick. Mine, ours called me on the phone.

It happens real quick. A person is out spending a half a day rolling out software.

Grant: Try to roll it back.

Spence: Now you got to roll it back, spending another half a day rolling the whole thing back.

So it has certainly enabled us to release and roll back faster, but I think as you and I were talking before the interview started, the whole notion of writing containerized software changes the way you think about writing software.

I was still cutting code when we made this change, and it certainly changed the way that I thought about writing software for the first 30 years of my life, or whatever it was.

Grant: The best practices and patterns for creating scalable distributed systems are just baked in.

If you're going to build it, it's going to be compatible with Kubernetes.

It's going to match those patterns, it's going to leverage those primitives, and we see a massive shift in terms of how software is being developed and deployed.

Ultimately though, I think the best part is going to be faster software releases, more reliability, and then that's going to create a whole lot of second factor effects and second order effects.

Spence: But also, it creates-- Now you have a control plane for your system, so you can start to wire in monitoring with Grafana or something.

You can wire in PagerDuty and you can log in and see what all the services are doing, and the days when we would just have a bunch of instances I guess you can roll all that yourself.

But that's an unpleasant task, and now you really have a control plane for a big multi-region distributed system.

Grant: Yeah. Because it's become the common way to do it I think there's actually going to be this really interesting piece.

Which is as everyone starts to adopt the same way to do this I think you're actually going to be able to hire engineers faster and onboard them faster into your product and your deployment without them having to learn some esoteric internally created system.

The ability to actually bring folks in using a common toolset across the industry to achieve these great results is actually going to have a profound impact as well.

Spence: That's certainly our story. We didn't have a dedicated dev ops person, so the team--

We have research scientist on the team now, and one of them in particular really likes Kubernetes a lot. He is all into it.

But anyway, we had people on the team learning how to do this and it was really quite painful because it is a specialized skill set.

There were not many people at that time who had mastered it, so it took us six months to find our first dev ops person.

But when we did, and he's a star in our company now, within two weeks he was like "Out of my way, you guys don't know what you're doing."

He owns our whole production system, whereas if we had a bunch of homebrew stuff that we had put together to run this system across regions, that would not have been possible.

Grant: That's cool. Then you guys also distribute this on prem to customers, right? You deliver--?

Spence: The other advantage that this had is that at that time we only had a hosted system, and now we have started working with customers who want to run say like an AWS VPC, or they want to run--

For some of our government customers, there are government regions of AWS that have different requirements.

Now we have bare metal deployments as well, so all of the investment to make the system--

To containerize the system there's still been significant work to support each one of these deployment targets, but each time we do it, it gets easier.

I don't know, we just wouldn't have been able to do that before.

Grant: If you were using-- First the challenge would be you were on Heroku, and there's no chance.

Spence: No chance, that's right.

Grant: And then the second option is like "OK. You're using AWS, but then you have some proprietary AWS service underneath the hood."

Spence: That's right.

Grant: It becomes harder and harder to do that, so the advantage with Kubernetes especially if you're leveraging some of these more portable components, is you can take it anywhere.

Spence: That's right.

Grant: That's cool. The other thing that generally helps with the security story too, so you're talking to customers and I often find that the vendor security questionnaires, there is really two parts to vendor security.

There's the core software development practices, and "Are you doing code review? How do you test for vulnerabilities?"

And that's going to matter no matter where the software is deployed.

Spence: And there's a policy aspect, "Do you have an incident response plan? Where's this system running?" Things like that.

Grant: Yeah, exactly. Then there's the operational side. It's like, "Do you have locks on your doors?"

Spence: That's right.

Grant: If people have access to your laptops and you don't have Yubikeys, there's all these things that they become very concerned about the policy and security of not just your systems but also your processes internally.

Just something that we think about a lot. Cool, so when you think about these enterprise offerings and how customers are using your software, are there any surprising requirements that you're seeing?

Things that you didn't expect but you're seeing from a handful of different customers?

Spence: I don't know if there are any surprising product requirements.

I think that customers expectations for stability and availability are very high.

The "Move fast and break things" ethos, which I actually find to be very attractive and was beneficial to us when we were trying to get product market fit or whatever you want to call it, was the right way to push.

We were just like, "Push, push, push."

Especially since we were trying different business hypotheses, you're like "Get the product in shape, put some feature out that solves 80% of the problems so that we can try to sell."

But now customers more value consistent and stable releases, and I thought that the move fast and break things was harder because there's a sense of urgency to get things to work. There's intense time pressure, but actually getting to a point where you can push things out consistently and more importantly, engineers, you only have to have a couple regressions before people start to get really nervous about pushing things and breaking the production system.

So we're actually in this trough right now where we're being really conservative because we don't have a staging environment that accurately reflects these sorts of phenomena in our production environment.

That partially has to do with cost, our production environment is pretty expensive and we can't replicate it fully.

So we're in the middle of trying to figure that out, but it's what our customers want, which is just like, "Don't break it. Because if you break it, I can't do my job."

That can be hard for our engineers to internalize sometimes.

So I tell them, "What is your tolerance for GitHub being down? How long before you blow your stack?" It's like--

Grant: Five minutes?

Spence: 15.5 seconds.

Grant: Yeah, exactly.

Spence: "OK, so now imagine our customers. This is their GitHub. Do you understand why they're so upset?"

Grant: It's one of those things where particularly folks who haven't been in enterprise for a long time or they don't really understand the critical nature of the work that's being done, it can be hard to really get it.

Because I think our industry is focused on that move fast and break things, blameless post-mortems, really like "Mistakes happen" and embracing that.

Which look, I agree, mistakes do happen. But we shouldn't sacrifice the rigor and use it as an excuse.

Spence: That's right. And by way, our team is doing a great job. This has been a real organizational effort over the last 12 months or so.

We've made a lot of progress, I'm simply pointing out the discipline and the rigor required in enterprise software--

This is going back to my early life when I was working on planes and stuff like that.

There you're doing things that are unholy in modern software engineering, like waterfall and writing huge requirements specifications, and doing significant software verification.

Grant: Informal verification.

Spence: Informal verification, exactly. And writing everything in ADA, which basically has static typing and basically will not compile unless all the types match.

There's a reason why those technologies exist, and I'm not suggesting you should go back to that, but when you're in the business of putting a plane together or something that is somewhere along that path towards that level of rigor is where you have to go the bigger the enterprise you serve.

Grant: It's complex. It's a hard thing to tell, and to help your team to understand, especially as we're using these emerging technologies.

Spence: We're using Kubernetes, which is breaking all the-- Our systems are written in tensor flow, which every release is not compatible with the previous release, so all the underlying technologies are moving really quickly and we're trying to suppress all that instability to provide this really stable application layer. That's tough, too.

Grant: Then the funny thing is you have a react front end and a single page app and some JavaScript error brings the whole thing down. It's like, "Wait. Why? The whole complicated part works perfectly and then you click the wrong link and white screen."

Spence: Yeah, exactly.

Grant: It's interesting, though. I think about this because there's vendor security questionnaires, but I haven't seen a vendor reliability questionnaire yet.

I think it's something that's probably going to happen more and more over time.

I think SLAs are part of it, but there's just so much now that we can be doing to build truly reliable software and the patterns are pretty established.

Maybe that's a thing that's going to emerge in the coming years, because software downtime costs real money.

Spence: It does, that's true.

Grant: When you think about like the future for Lilt, you were the subject matter expert, the CEO and the CTO for the first few years.

I want to say three and a half or so? Somewhere in that range.

Spence: I had I had a great team. That was the key part, I think.

Grant: Sure, you had a great team. But you were leading each of these functions for a while. Right?

Spence: That's true.

Grant: And that's a hard thing to do as a founder. Your co-founder isn't full time at the business too, right?

Spence: That's correct.

Grant: Which can also be hard, there's some amount of - - Just a lot of weight on your shoulders. Right?

Spence: I think if it was a different person, maybe. But John and I have worked together the same way for a long time and it may look non-standard externally, but it works for us.

Grant: Yeah, sure.

Spence: I don't know what I'd do without him.

Grant: Yeah, totally. I get that. I guess my point is you're now making this transition where you're bringing on some other leaders.

Spence: Yeah, that's true.

Grant: You mentioned you've brought on, you're really just taking your first step back from managing the software team day to day basically, right?

Spence: Right. That's right.

Grant: One of the hard things is finding folks that have that experience to come in and help lead and be executed at this level, because you're a startup but you're also trying to accomplish big company things.

So what are some of those areas where you're looking for great people?

Spence: We've got some great engineering leaders in the company now, that's taking some time. That is very promising.

We're still looking for-- I'm also the acting head of our services team too, so we're looking for a services leader right now which will be a partner to both sales and to engineering.

Grant: Then by that do you mean actual professional services, where you're helping to set up the systems?

Spence: Managing our translator teams.

Grant: Managing translator teams? OK.

Spence: Yeah. We're going to have a separate function in engineering.

We now have more requirements for onsite support and more common professional services, whereas for us services has always been managing this labor pool, our community of translators.

But we do have increasingly requirements to with these on-prem deployments of supporting customers wherever their deployment is.

Grant: Then this idea of someone's operational and can come in and help manage a large talent pool and help them accomplish the "Human in the loop"part of your job?

Spence: We're still looking for that leader. The other functions are starting to consolidate.

Grant: Yeah. This is one of those businesses that's-- A lot of my time is spent in infrastructure software, like software for software engineers.

And I think this is interesting to see because the customers is very different, and it's also not marketing technology.

It's not like there's not a huge category of other contemporaries that are all doing related things, so it's a smaller talent pool to find folks in.

Spence: Especially engineering-wise, the number of people who have built and managed core machine learning products, most of those people are in the big companies.

It's very hard to find that in startups. You can find it some places, the nearest analogs that we found are in security ad tech.

Like, large scale distributed databases and stuff like that.

Grant: Sure.

Spence: You can find people who have managed systems of this complexity.

Grant: Yeah. One of my friends, Zach, runs a sales team here and he's described it a couple of times as really more about making the world's information available.

Spence: The mission of the company is to make the world's information available to everyone, irrespective of where they were born or what language they speak.

Grant: Yeah, it's a compelling mission. There's something behind that. What's the backstory? I'm sure there's some more detail on that I'd love to hear.

Spence: John and I both, that's why we started working on machine translation, was that we were both inspired by experiences in life to work on information access.

They're two choices, or they're two solutions, which are to teach everybody English or some lingua franca or build technology to make information available.

I think the latter is more attractive than the former.

Grant: Yeah, it's interesting.

I'm an optimist so I always talk about how I'm excited about the future of the world, and I would say "Now you can grab the world's information out of the air around you basically for free. Nearly free."

But the reality is, it's the world's information probably in English.

Spence: Yeah, that's right.

Grant: So that idea of "If we're going to truly make it available to everyone in the world, we need to translate it so that everyone can access the world's information in a way they can understand it."

Spence: That's right.

Grant: Yeah, that's cool. Great, so I'd love to give you a few minutes here, if you are up for, how do you describe--?

I know we've talked about it a lot, but just the quick pitch that you normally give, however long you want to talk for?

Spence: Sure. What we do is we provide a full solution to large institutions for translating information and providing their information to their customers in the language that they prefer.

The word "Solution" is important, because to do this you need software to manage the workflow and to make the work efficient.

The translation is not a solved problem with technology, so machine translation best embodied by Google Translate which is hands down the best general domain system that exists, and is what certainly inspired me, and I can speak for John too, to work on this technology.

It does not provide what one institution needs, which is a certificate of correctness. The only way that you can do that is to have a human involved somehow.

The whole crux of the problem is to make that human intervention efficient, so when we call it a "Solution," we mean that we can provide the full piece of software to manage this process along with the people to staff the production process.

And typically what businesses do now is they'll buy workflow software from this vendor, and they'll buy people from this vendor, and they'll buy a quality control from this vendor.

Some companies are dabbling with machine translation now so they might get a Google Translate API key, and then they're just putting together this Frankenstein mess that costs too much and doesn't result in the outcomes that they want.

Or maybe it does result in those outcomes, but it's really expensive and it takes a lot of labor on their part.

Now if you rethink the way the whole process works, but you assume that there is a machine translation system at the center of it and you start to build around that, you arrive at a different solution than if you assume that the human is at the center of it and you just add some things onto it.

You arrive at a different business model, you arrive at a different technology stack, you arrive at completely different outcome.

What's interesting for John and I is that we have been building, I don't know what we're on, the sixth or seventh generation or something of this that we've worked on.

It's still the same thing, it does the same thing that it did when we wrote the first prototype in 2012 or whenever it was.

But the way we talk about it and all the people that sell it and talk about it and the way it's serviced and all the workflow stuff that goes around it, that's all different, but it does basically the same thing.

I think the hardest thing for us to learn is that the way you market things and sell things and talk about them and price them and package in them and start working with a customer as a partner, "Will you execute with that partner?"

All of that stuff, that's running a business. Not just building a piece of technology. We could have learned that sooner.

If we did it again, we would be quicker. But it's been quite a learning experience.