Why Duolingo Worked (with Luis von Ahn, CEO)
01:03:22 Share

Hello, acquire listeners. We have an excEllent A C Q two episode for you today. We are here with Louis von on the cofounder and CEO of duo lingo.

Luis was a math professor at county melon in pittsburgh from two thousand and five to twenty twelve he previously founded, which i'm sure many of you have filled out hundreds or thousands of times that sold to google in two thousand and nine. He has guta alan and immigrated to the united states studying mathematics at duke and later earning his PHD in computer science from caring melon. Welcome to the show the is.

Thank you.

You are having me. We are very excited to this together. I think duo lingo is a completely fascinating company. And rather than me saying duo lingo is a, why do we ask you what is do a lingo?

Well, we're most well known for an APP that teaches languages, that is the most popular way to learn languages in the world, that is the most popular education APP in the world. We have about one hundred million monthly active users, were also known for unhinged ed. Yet hold some Green out that does weird stuff on tiktok.

It's the mascot of the company. You know, that's basically what we're doing, what we can to teach people languages. IT started because we wanted to teach people english.

And turns out english is transformational in the U. S. I guess to, if you don't know english, the U.

S, you really should earn english. But in non english speaking countries, english is is pretty transformation ally. People who learn english can just make a lot more money.

So we started with this very mission driven idea that we could teach people english all over the world. But if we are going to teach english, may as well teach all the languages. We started teaching other languages. At this point, we are the most popular way to earn .

languages in the world. amazing. So my first question that i've been listening to another podcast and doing research on the company preparing for this, why has duo lingo worked to such an extreme scale? You're nine billion dollar publicly traded company. Why has IT worked when so many companies in the language category in the past have been this small, fragmented sort of subscale companies?

There's a number of reasons. But if you ask me for one single reason is that we understood something early on. The insight is the hardest thing about learning something by yourself is stay motivated by a margin.

And so what we've done with doing was really, really have made IT so that you want to learn. And I think that's what has made us very popular. Most education apps that try to teach something concentrate mostly on the learning outcomes.

They try to actually teach the thing, and that makes sense. However, that's not the hard part. The reality is you can learn anything with a book.

You you want to learn quantum physical. You can do so with a book is just nobody is doing IT because IT requires reading a quantum physical ook. But if we can turn that into something people actually want to do, that can be really powerful.

And I think that's the biggest. And I mean, there's other things that we've done. We became really good at product. We have really good design. There's a lot of things, but ultimately, just that we understand that we need to get people to want to do IT.

Alright, so what are some of your key levers in making people want to engage?

When we started to a lingo, which was link twenty eleven or so, we started working on IT. We were building a program that thought you languages, and I was with my co founder. My co founder is a native swiss german speaker and am a native spanish speaker.

And what we decided we were going to do is I was going to make the first spanish course, even though I don't know how to teach spanish, but I do know spanish. And he was going to make the first german course. That was really stupid.

I thought german and swiss german were the same language. Turns out they're not. I thought they were like british english and american english.

They're not. It's like a different language. Anyway, he does happen to also know german, like actual hot german. So where he made the german course, I made the spanish course, the goal was that we were gonna n each other er's language.

But we brand to this problem that we would go home, and then the next day we would come to the office at nine, and I would ask him, you did you do your spanish lesson? And he would say, no, man, so boring. And I had the same problem. It's because when we started, we were making these lessons that were like thirty minutes long first table, and there was no game fiction that said, and we couldn't even get ourselves to do IT. So this is when we started really trying to add all the mechanics .

to get people to do IT. Was this in anyway connected to your research?

Or was this a complete size? Well, I kind of was. He was my P H.

D. Student that was a professor, and we were trying to find A P. Hd thesis topic for him. That's what we were doing. We didn't even know this was going to be a company.

I mean, we cannot in the back of our our mind, thought maybe this could turn into a company. We were in super sure that was the idea that was kind of connected to my research, but ultimately was just try to and teach you something with a computer. Your research was around crowd sourcing, right? My research was around crown sourcing.

Yes, and this was just more about trying to teach something with a computer. So we were doing that. But this is when we learned that we wanted to make IT more fun.

And so we started doing a number of things. I think the first thing is thirty minute lessons. No good.

You want to do three minute lessons. So that's the first thing with IT in three minutes. That's very important. Even if ultimately people do spend thirty minutes on your APP.

Making each chunk short matters a lot because that means you can do them while your waiting for the bus or while you're literally go to the bathroom thirty minutes just too long. So you have to be able to chunky. So we started chunking, and that's important.

So we made the lesson experience be something that was quite palatable ble. Three minutes there's a progress bar that you get to fill. And then we started adding all kinds of little dopamine e hits.

Like when you get something right, there's like a little explosion, that kind of one thing. Then over time, we discovered a number of other really important levels. One super important level has been the streak.

So we have a streak. We've had a streak for like twelve years, basically. And the ideas, you know, the standard with a strict, if you use IT for seven days in a row, you have a streek of length seven.

And if you don't use IT on eight day, IT goes down to zero. So it's just a counter of how many days you have done IT in a row. This is super powerful at this point.

We have I think i'm going to get this number right. I think we have eight million people who are daily active uses. We have a street longer than three hundred and sixty five.

So we have eight million daily active uses that haven't missed the day in the last year or more. So more than three hundred sixty five. This is very powerful trick is .

very powerful that around eight percent of your users.

eight percent of our active users have not missed a day in at least a year. That's very powerful. IT really is another thing that has been real powerful.

notifications. You know many apps abuse notifications where they just a notify you like crazy IT turns out of the view, teaching something to somebody. There are a lot more OK with you sending them notifications because people think, well, this is good for me. So we've taken an advantage of that. While we're not spammy, we do send notifications probably have a higher rate.

And we could if we were not teaching you something, we have a very sophisticated system that tries to figure out is an A I system that we have trained over the years that basically try to figure out when to send you a tivo, what to say in IT to get you to come back. That has been really successful. And then the other thing that has been pretty sucessful, that we've gotten just a lot Better over the last few years, is a social component where we have leaderboards leader boards.

By the way, there's a whole science of how to them. Well, we've got them very wrong at the beginning, but we are much Better now. So, you know, we have leaderboard. You can have streets with friends now. You can have quests, friends.

And we should say this notion of game fiction, I mean, this was like a real hot word. 2 color， twelve to fifteen. yeah. And you started the company in twenty eleven. So you were kind of blazing the trail on a lot of these things. The idea that you could bring game mechanics to non game experiences, it's pretty interesting reflecting back on that time how that was the buz word in every VC pitch that we were seeing that you really did build a public company on that concept.

Yeah, for some stuff, game fiction is not all that useful, particularly things that you anyways have to do. So for example, a smaller part of our business for dual ingle, we have this thing called a dual ingle english test, which is about ten percent of our business. It's a standardize english test like tofu.

It's like tofu. It's a competitor to tofu is a test that people have to take to get into a university, for example. That thing you can game fy, you can not game fy.

IT doesn't matter. The reality is people have to take IT and y'll go through IT for something like education. I think signification ends up really mattering a lot because it's all about motivation actually keeping you there. So I think this was just a very good use of game fiction in the end.

Do you have a sense of if you've expanded the market for people learning a language, who wouldn't have sought to learn what otherwise massively .

IT depends on the country. But in the united states, for example, eighty percent of our users were not learning a language before dual lingual.

Typical american .

like was not .

learning in yes.

yes, but they are now it's a similar in the U. K, it's a similar number in the U. K.

There are other countries, particularly countries where people are learning english there. We probably haven't expanded the market as much, but in english speaking countries like the U. S, U, K, australia, we've expanded the market by a lot.

Did you encounter when you were fund raising a lot of push back that, hey, this market isn't that big. I imagine it's pray hard to solve the thesis. Our product is so good that we will massively expanded and of the market.

Yeah, we did. Our first fund race was twenty eleven, union square ventures. At the time, union square ventures was very hot. They are there was a big deal anyways.

We went there and we were so fortunately that they found that as bradburne was our partner there, he got into our board. We became good friends that are many years later, he called me. We didn't think you are going to succeed at that.

We just like you, and we thought you were going to pivot into something else and maybe you'll do something, but we actually didn't think you've going to succeed because we didn't have a product. So that was the first round told the serious a by the time we reached the series. B and we raised canada results book for themselves.

We aren't making any money, but we had a lot of users, especially if we had only launched like a maybe two years before that or something. And we really already had millions of years. So they became a lot easier when we had I don't know how much product market fit was, but at least we had some amount product market IT.

I'm curious what your thought process and discussions were as you are starting the company. But even before then and you're looking for a PHD, were you always planning to be mobile first, mobile native? Because that was right around that generation when I switched .

from like you're exactly right in twenty eleven, almost no company was doing mobile first. Everybody was like website first because most traffic was on the web like a big laptop, even p when we started, we thought be a website. But relatively early on, really, maybe in the first few months, we realized almost everything is going mobile.

And we had this insight, which was we're very fortunately that we had this insight that at the time, most companies, they were website first and they had a companion APP like banks. You could go to the the main website, but in the companion APP, you could check your baLance. We couldn't do anything else or what, but it's just like a companion APP.

We had the insight that you know we can actually put all the functionality of the website in the APP and so was not a companion up or anything. And as soon as we launched, we first launched the iphone up. IT overtook the website traffic.

Within weeks, we had more traffic on our iphone APP than on the web APP. And so right then is when we decided we were gonna mobile first. And at this point, my god, I don't know that it's ninety five percent .

of our traffic of as this was happening. Yes, because like the analogies are so close here we were .

and by the way, we were very fortunate with incident and and I also think we gained a lot from this because know we were not the first language learning APP. There were others, but I think we were the first good one. And it's because before us, what people were doing with apps is they were basically making websites and making them tiny is I got a tiny little website, but they just going to a strong everything into a mobile screen.

And when we came up with a mobile APP, by the time we came, we were watching things like instagram. There was another path, was a time. So we were watching all these apps.

And we are like, this is how you sign an APP, not make a website and shrink IT. You actually have to change all these things. So I think we benefit a lot from the thinking of let's make an actual native mobile thing, I suppose, to administration website. We are very fortunate with that.

I'm obsessed with this misperception of market size question. I'm sure you get pitched a lot of ideas now. Many people probably come to you with the idea. Hey, just like duo lingo, this doesn't seem like a big market, but actually, we're going to build a big business in IT. What's a good heroic for when that is possible?

Vers, not. I don't know what a good heroic really is, but I can tell you this. I think language learning was massively under appreciated in the united states because in the U.

S. People don't have to learn language. The realities, if you already speak english, english is the the monster language in the world.

That's the big one. I grew up in a country where you had to learn english if you wanted to be mildly successful. This is true in my case.

Had I not earn english, I wouldn't have come to the U. S. And I would have definitely not succeeded the way that I did. And Michael founder was also not born in a english speak country.

I think when you grow up in a country that is not english speaking country, I think you realize, yeah, you may think that the market is not huge, but pretty much everybody is either learning a language, in particular english, or once to learn english, and everybody pretty understand. So at some point you just realized, well, boy, billions of people want to learn english. So IT just has to be a big market.

There has to be a big needs. So I think that was the case for a language learning that, you know, I don't know for the other ones, what we do inside do a lingo. When we're looking at opportunities, we definitely look at just how many people are either doing something every day or close to everyday.

So this is why we decided when we're trying to expand to other subjects like we do math, music, we went for those because those are things that literally billions of people are either doing or want to do. And we didn't go for something like, you know, you could go for chemistry or something. Love chemistry is just there's probably not a billion people learning chemistry or wanna to learn chemistry.

What's your early signal on duo lingo for x? You know, you mention math, there's music. Did you start with the very best use case for the type of product you build? Or do you feel like or actually that was a local maximum? There may be even Better products that are more well suit.

There's a lot of companies. I mean, I get the ads all the time, unlike my instagram or something, twitter or whatever that says, you know the dual lingo for whatever. There's curious want to do lingo for anger.

And I don't know they mean that you like teachers how to get angry or not? I don't, but anyway, I get a lot of ads. I said the dual ingua for anger anyways.

My sense is that there are probably other things that are pretty good, but it's hard to compete with language learning. I think matheu music are gna be big. I don't know if they're going to be as biggest language learning, but there are gonna be big.

But the reason, I think, is hard to compete with language learning is that not only are there a lot of people wanting to learn english particular, but also language learning as a few things that are good. One is that is one of the few things that people want to learn, both in school and outside of school. So most things people just going to have to learn, like in school math, is an examples.

Not many people want to learn math s outside of school, but language learning, if you ask the average person walking on the street, you know, would you want to get Better at spanish? Most people like, yeah, why not? So I think that's a big thing.

Another big thing that happens. This is in contrast with music. Music has one big problem, which is you need an instrument and that's a big deal.

I mean, unfortunately, an instrument is two three hundred dollars with language learning. You don't need anything that there's a lot of things that I think work really well for a language learning. I don't know. It's the best use case, but it's a pretty good one.

yeah. So thinking about the language learning business you mentioned earlier, english is the number one language that people want to learn. So most of your usage, please correct me if i'm wrong, is outside the U.

S. People wanting to learn english, but most of your monetization happens in the U. S. I was working if you could just tell us a little bit about sort of how that evolved in how you think about IT.

yeah. If you look at the numbers, about forty five percent of our activities sers are learning english. English alone accounts for forty five percent of our users, and that's mostly outside of the U.

S. In terms of users in the us. About twenty percent of our users of active users are in the us.

The other eighty percent are outside. There is a revenue, but half comes from the U. S.

And the reason for that is because you know the U. S. Is pretty large and wealthy country.

But the other thing is that important to understand this is the people in the U. S. Are very accustomed to paying for digital descriptions.

But we're finding is that in certain countries, the U. S. Is ahead of most everybody. In certain countries, people are a custom to dial in. Others, people are not or they're really against them.

And these are countries were even a spotify or a netflix are just not as penetrated as they would be in a place like the us. I think that's a big thing. We just have a lot more propensity to pay in the U.

S. And wealth your country's. The other thing that happens with us, as you know, we have a really good free version.

In fact, if you look at our monthly active uses, only about nine percent pay. The other ninety one percent do not pay us. And it's because our free version is so good.

And what we're finding is that in poor countries, even if you bring the Price down, a lot people have this mentality, that is that they won't pay on list, they have to. So even if it's like fifty cents, like super cheap, unless they have to, they want pay. Where us in the U.

S. People are a lot more willing to pay for convenience for things like, well, well, you know what, I don't want to watch ads, so what whatever i'll pay. We're not seeing that type of behavior in a country like vietnam or brazil or something like that. I mean, some people pain those countries, but it's just much less than the U. S.

And you do have an ad supported to your .

right yeah with this, the free tier IT has ads at the end of a lesson, but the ad load is prey. I means basically you get one skiable ad at the end of a lesson. So five seconds of ad after about three to four minutes of of thing, it's pretty light. yeah.

And the other thing that I think is important as listener sort of understand this part of your business, when did your product launch and when did you charge the first dollar of revenue? Oh.

we launched in twenty twelve, but when we launched, we made no money. We didn't have ads and you couldn't pay us. I mean, simply, we made no money because there is just no way for us to make money.

We we just didn't put IT there. And that was true up until about twenty seventeen. We started thinking about monetising in twenty sixteen.

What happened was we had just raised funding from, at the time I was called google capital, but is not called capital g. We have just race funding from them. And I don't remember the valuation is about a half a billion dollars valuation with no revenue whatsoever. And the partner from their lay last thirty flew to pittsburgh, where we were, and, you know, he got a strong and he said, you ve got to start making money.

And of course, as a founder, you clearly knew the secret that the only thing Better than making a lot of money is making no money. Everything in between is not good.

Yeah, exactly. No, that is the secret. And and so went to the office the next day, and he was pretty interesting because for the first several years, we made no money and we were even really trying to make money.

And everybody that we were hiring was very mission driven. We were hiring all these people that were that loved the idea of giving free education. So when I came to the office and I told everybody we have to figure out how to make money, people were pretty shocked and they were like, but why? But I said, well, cannot have to pay your salaries. And they said, you've been paying our salaries this whole time.

The problem so far, you.

why can you just continue doing whatever of the help is you've been doing? And I had to explain, you can't raise funding for here.

We're going to do we'll do a dual lingo session on late stage capital.

And you you know what? I was crazy good about six months to convince ce, the employees of the company that I was, in fact, not evil to make money. Eventually, people are convinced.

And I think the main way in which people are convinced, I think everybody understood that if we made money, we could invest more in the mission. And I think that everybody was going, okay, okay. So you mean we can still put most people to work on actually making a Better product that's supposed to making money.

And I say, shuter, that's what we're going to do. Today's day remains to him the fraction of people that are working on making money at the companies, relatively small compared to the rest. I took about six months, convince people that we should do that. The other thing that was complex at the time, we weren't sure how to make money, a way that respected the fact that we wanted still to be the case that most people could learn for free.

You know, the time spotify was kind of growing and said about IT IT was not not super clear that this premium model worked this free me mode, which is like you can use this for free, but you may have to see some ads and if you don't want to see ads, you have to pay. IT wasn't clear that we could make that work. He turns out you can make IT work so great.

IT worked really well. And the fact that that worked was really good for us because it's very missioned. Basically, you can learn entirely for free. You can learn without ever having the poos, but if you want to turn off the ads, you can pause us. That's very missioned for us, but we're fortunate that work he really was not clear back then.

IT is so funny. That's like to such a basic assumption now but even twenty, twenty, eighteen, youtube, you couldn't pay them. You could only watch ads. Netflix, you could only pay them. You couldn't watch ads.

Actually, David, I think google contributor did exist.

I don't even know what that was.

IT was the processor to youtube premium and IT worked across other properties where you can basically pay the amount that google would make on you from all the hours that you .

would see the there's no way I think I might have .

been except for Edwards. I don't know. I should get my facts rate on this. I was a pain goole contributor, customer. I mean, IT says a lot that I was called contribute.

That's funny. Well, again, for us, IT was not super clear that this was going to work, but IT worked.

Was IT strategic at all. Was there in network effect component? Like is IT good to not put any inhibitors on your growth rate because more users but gets more users or something like that?

I am a huge believer that the recent that we have been able to grow so much is because we did a monitise early on. Now I cannot recommend that to every single company. I mean, for some companies, you probably should monitise early on.

But in our particular case, what happened was this, we launched because we weren't making any money. We also couldn't spend money on marketing like we couldn't do performance marketing because performance marketing is this thing where you like, you can pay google thirty books and then they send your user. But to me, that seemed really done to spend thirty box to get a user. When we were not going to make any money off of that user.

you be more than thirty dollars from that easy.

yes, or whatever IT is, you wanted to do more than that. And so we couldn't spend money on marketing. We weren't making them any money, and we had a bunch of engineers.

So what were we going to work on? And the only thing we decided to work on was teaching Better. And in increasing user retention, that's IT because as the only thing we could work on.

So we spent from twenty twelve to twenty seventeen basically optimizing retention rate for users, and we optimize that a lot. I mean, for example, we went they want retention. So that's the retention of people when they show up what fraction of them come back the second day with the next day when we first launched, that number was thirteen percent.

As that you lose eighty seven percent of your users the day, the day.

Well, they may come back. Another eighty seven percent happen to not come back the day right after he could be that they come back later. But that's what we saw. We ran. Who knows how many A B tests over this period time, but by twenty seventeen, that number was, I know, a probable about fifty percent and maybe misremembering exactly the numbers.

But we were able to drive all of our retention numbers by a lot because as the only thing we could work on, had we not done that, we probably would spend all our time optimizing our ad pipeline and optimizing all kinds of weird stuff that ends up making your money. But I don't think helps you grow that much. And I think we were in retrospect, this was a very smart move at the time. I don't think we are thinking how smart this was.

just kind of did IT IT gave you a durable asset? I mean, the fact that a product that, that ensures that you have fully maximized to the potential for a user to stick around IT means the user base, if you've built and the machine by which you eventually do make money is far more durable.

That's right. And I believe that now of course, I believe all of this, in retrospect at the time, were just like I don't make a thing that people .

like I was also were raising money to have a billion evaluation.

So this was when money was free in some sense, just like here have more money, but IT worked. okay.

So you joke that you weren't doing any marketing. In some sense that's true. In some sense not. You have the Green now. You've had a great brand forever. How did a couple of p hd from carnegy melon of all places come up with a cutely Green now and build? I don't think of CMU as wonderful as institution as IT is as like the top design and marketing place in the world.

They haven't have a really good design program, but looked this evolve over time. We did know one thing. We did want a mascot early on.

We were like, look a maScotte. Somehow this will help us. We didn't know exactly how. We didn't want a scot.

Eventually we chose an out because of, in western cultures, res, at least, is a symbol of knowledge. He attended up being Green because my co founder hates to call her Green. And I thought IT would be funny. IT was Green.

That's why I was Green. And now your whole office looked behind you.

The first version of that was not very cuddly, or, I mean, I was kind of not great looking, honestly. Over time, IT evolved. We hired some amazing illustrators.

We hired this, this guy who still work for us, great hartman, who's in our head of art now. He just made an amazingly beautiful hour over time that that evolved. And so that's one thing.

We just made a really good asset that looks good. And then the other thing that happens is is started getting a personality. But the personality was a little bit accidental.

The first way, which he gets, the personalities the out was actually not in the APP at all. At first, the only place where people saw was the APP icon, which is the kind of a very close of version of the house face. And then also, when you get a notification, IT has the abbie, so IT looks like IT comes from the out.

So the first place where the personality for the l came up was basically what we said in the notifications. And we started saying some weird stuff in the notifications. And I think people started giving me a little bit personality because that then another thing happened with the notifications.

At some point, we decided we can be super me. So we're gona stop sending notifications after five days of activity. If, if you don't use the up five days, stop. Then IT occur to us, well, we gonna stop sending on notification.

We may as well tell you get the credit for IT.

Yeah so people know where good guys, this is what we thought. So that last notification, the fifth day we wrote IT is just said these notifications do not seem to be working. We're going to stop sending them for now. That's what I said.

Well.

well, we didn't know this. We expect that. Turns out this got people to come back a lot because they thought that the bingo was giving up on them. It's like your mother. It's kind of like guilt tripping you.

Because of that, though, I started giving the right to kinds of memes on the internet about like how the how was obsessed with getting you to come back and he would kill you if you didn't come back. But that, you know, that was the community. We didn't do that.

After that, we realized that we that we could push on those memes. And so we started making fun of ourselves with those memes. And that has worked really well. And at some point we got the idea of getting an all suit. A suit that somebody could wear.

At first we thought I was, he was going to be good for recruiting events, like going to try to hire people at universities and downs IT would show up, turned out you could start putting somebody and outside to do weird stuff in tiktok videos. And that gets a lot of people to watch IT. And so, but that was a little bit accidentally discovered. In fact, the first person who started doing this was a very Young woman is sarria, who, when he proposed that, I thought, this is such a dum idea, I was obviously completely wrong. IT turned out this is given us a lot of prominence in in the world.

Have you tried to quantify I mean, this is like a core pillar of dual single business strategy. At this point, if you tried to quantify the business impact of having all this or rent media.

yeah, I mean, we have our belief is that about fifteen percent of our users come from that on any given day. Roughly fifteen percent of our new users come from .

a Better some sort .

of our thing. The majority of them come from people telling the friends about IT, but about fifteen percent come from this weird stuff.

And just to multiple some other numbers together that you've talked about, if he really is fifteen percent and you have a hundred million users, is what's the value of fifteen million users? Well, if it's actually at at thirty box a pop would be the paid cost to acquire them. That's what four hundred and fifty million dollars.

Well, you're only taking the active .

ones thirty dollars on google to get a user I D.

A user that may not stick around. So for a hundred dollars of the most valuable .

all in .

history, I don't know that through, but it's been really good for us.

It's fascinating. We want to think our long time friend of the show, venta, the leading trust management platform, venta, of course, automate your security reviews and an efforts. So frameworks like soc two I saw twenty seven o one gdpr and hip a compliance and modeling ing vantine, takes care of these otherwise incredibly time and resource training efforts for your organization and makes them fast and simple.

Yeah, fanta is the perfect example of the quote that we talk about all the time here and acquired. Jeff bases his idea that the company should only focus on what actually makes your beer taste Better. I E spend your time and resources only on what's actually going to move the needle for your product and your customers and outsource everything else that doesn't. Every company needs compliance and trust with their vendors and customers. IT plays a major role in enabling revenue because customers and partners demand IT, but yet IT add zero flavor to your actual product that IT takes .

care of all of IT for you, no where spread sheet, no fragment to tools, no manual reviews to cobble together your security and compliance requirements. IT is one single software pain of glass that connects to all of your services via s and eliminates countless hours of work for your organization. There are now A I capabilities to make this even more powerful. And they even integrate with over three hundred external tools, plus they let customers build private integrations with their internal systems.

And perhaps most importantly, your security reviews are now real time instead of static, so you can monitor and share with your customers and partners to give them added confidence.

So whether you are start up or a large enterprise and your company is ready to automate complaints and streamline security use like vana seven thousand customers around the globe and go back to making your beer taste Better, head on over to vantage a combs required and just tell them that ban and David sent you and thanks to friend of the show, Christina anta CEO, all acquired listeners get a thousand dollars of free credit vana a com slash acquired okay.

I'm going to keep coming back to my like, what's a heroic type questions start up comes to you with the idea that they're going to run a similar strategy. Any advice for when I could work or not? Or is this peer lightning in a bottle? I have no idea what IT will work, some stuff.

I think you can reuse this a few pillars in a strategy. One thing that we haven't talked about that I think accounts for probably half the value of the company is our obsession with A B testing. We have ab tested our way to where we are now, and we have probably run want to make this up probably about two thousand and eighty six per year of all kinds of things.

And so we are just built the company around our entire product organization. That's not just product managers, engineers, designers. The whole system is built around shipping A B tests as fast as possible.

That has worked really well for us. And I would say you should do that everywhere if you have a consumer product. I mean, you shouldn't be A B testing stuff if you have like A B to b saying whatever, do something different.

But for a consumer product that has a lot of users, you can really A B test your way to improve almost any metric you can that you should do the other thing that I would give advice to his hand, really good people that you should continue doing in terms of the specific strategy of having a premium thing that you don't monitise for a while. I'll tell you one of the reasons why that may not work anymore. We have found ourselves, but we have seen at all over the place.

We were fortunately that we put out an APP in two thousand twelve. The APP market was not yet set in stone. It's pretty close to set in stone these days is not a hundred percent.

But if you look at what of the top fifty apps globally for the year this year, verses last year and you compare they are pretty similar and maybe one or two changed, but they're pretty similar and it's similar to the year before and it's similar to the year before. It's just at this point is pretty hard to enter. And I don't think that the strategy that we employed in twenty twelve would work today because the APP market is pretty set in stone. I mean, it's just gonna.

What about the strategy of something clever earning you hundreds of millions of dollars of sort of free here to media?

It's great if you can do IT. You know, of course, a lot of cleverness in this that many employees are doing did, but I think we were luck in a few ways. I think the fact that we are an education company gave us license to do some stuff that many other companies may not have been able to do because it's kind of hard to say that language learning is bad for you.

So that means that we can do risky stuff on tiktok. But if a company that is trying to do like, I don't know, like some sort of fin tech company or something, I think think is much easier to accuse them of stuff. And I think that helped with us that just come on and everybody where the good guys, I think we were able to get away with a lot of stuff because it's hard to say bad things about language learning.

right? If i'm coming at you with, you could argue it's a fair interest rate or you could argue it's a predictor interest rate. If I have a maScotte that sitting there with a knife saying use or at more, it's it's not onna go. Well.

exactly. And I think we've benefit a lot from the fact that ultimately we're doing IT for education. So people are OK to to take some of this right?

Fascinating.

by the way. You know another place, when we talk to other companies, every company wants to talk to us not because they learn from our book, because that I know you guys, yeah, you guys are kind of fun. This is good.

And nobody thinks of us as competitive well as great for us. And it's because almost every time we talk to whoever fancy person at random company is my daughter loves you. I think we've benefit a lot from that.

which is crazy. How many other ten billion dollar companies could that be said for like a year or ten billion dollar company? Almost you d think like defacto, you're gonna people who are your model enemies again.

because I was in now yeah yes. So an interesting .

comment on competitors at this point. You probably don't have a whole lot of meetings about legacy language learning companies and saying, how can we acquire users versus them? Who do you think about as competitors for your users?

I mean, the way we see IT as we compete for time. And so our competence are instagram take talk ts that that's that's we compete for time. And and by the so we're losing, we have much less time spent on our APP than tiktok hh or or instagram IT set.

But when we look at our users and we see that our users maybe stop using the link or something we ask them know to stop. The most common answer is like, guy, you know what? I'm spending more time on pick your random social media site. I'm spending more time on instagram. And that is that a competitor is whatever takes your time.

Will you play into the seed of something that we talked about like one minute in this podcast? But every person I talk to who's ever interacted with dual link of former employees and other podcast you have been on is all about experimenting A B test, being data driven. There has been lots of inspect on this for anyone who wants to read excEllent blog post about the know, the AI behind your push notifications and how all those systems work. One high level question for you. How do you as a leader, baLance a cultural tenant being a data and experimental driven versus instant driven.

we're actually weirdly both. I'll tell you where this comes from. We realize early on that pure experimental can lead you to really bad places. If you can just don't think about IT at all, you you just gna up to my symmetric. I can need to be really bad places.

And one of the best stories that I heard about that, and we repeat this a lot to do lingo, something that we learnt from the founder of group on who happens to be from pyper. K, and we're locating pitzer, he happens to from this perk. Yes.

we're huge, descript users. yeah.

Is a great guy. He told us the story, which you not going to tell IT. And at this point, product has changed a little because I I probably remember now what is in the law of dual linga supposed to exactly what story he told us. But you know, he told us a story about emails, that group on that per day.

So, you know, group on, basically you got a deal and the ideas at the beginning, you could they could only send one email per day with one deal he was strict on its like only send one email per day because more than one email is spammy. At some point, some enterprising product manager shows up and says. What about if we send to and amazing just no way like, no, we are only going to stand one.

And then this park manager gave this amazing argument against IT IT. We're not trying to launch to. We just wanted know are you are against knowledge, and of course, nobodies against knowledge.

So so you know, he is that in? Fine, fine. And so they tested to, guess what happened when they tested two? Two is way Better than one.

So they started sending two. And I like, what if two is Better? Test three, turns out three is Better than two and four is Better than three is set.

And the A, B tested their way to, like, I don't know what, number seven, eight or something, until the channel stop working because people marked at a spam. And then he died, the channel died. But so this is the thing about A B testing area, like you have to, you have to really do the common sense thing above the A B test.

And so what we do a do lingers. We have, we have this two step problem in the first one is before you even ab test something. And by the way, here we are against knowledge. There are certain things we just don't want to know, and it's because it's too tempting to do IT. So the first step is before we do.

And at the test, we have this process called product view, which I know soon happens in many companies in case every single change that is to the product view, basically set of leaders in the company, that includes me, this about five people. And we look at every change and we have to agree for IT to go out. We stop a lot of things before going out to prevent this type of stuff.

Here we use intuition and just common sense. But like, no, nobody wants to receive thirty two thousand notifications. That's just dumb, even if the A, B tests. So I think that has helped us quite a bit.

Do you have an example of way in which you've been anti knowledge in the past, something you decided .

you didn't want to know? Oh yes. I mean, pretty recently, somebody wanted to know what happens if you put an ad that takes over the whole screen when somebody opens the APP. Like a product manager wanted to know that. And we are like yp, but we don't, particularly with monodist ation.

It's so tempting because what happens more to you run the experiment and then IT says something like if you launched this experiment, IT will make you fifty million dollars a year, and you look at on you like this, nice. And then the finance team figures out that this happened like somebody from the finance thing looks at this experiment, suddenly the CF is in my conference room or something, and he's like, you million books a year, you to launch that. We tried not to know the answer to many of these monetization experiments .

when you wear your CEO or your your head of product, that these things are sort of obvious. And and what's underneath IT is this implicit assumption that there is a future dollar value associated with treating the user with respect and having a feel good association with your product. That is basically like all of the potential future dollars we could earn are jeopardized by doing this. It's we're polling forward revenue.

if very hard to quantify, but it's very hard to quantify .

complicated expected value calculation on IT. yeah. Have you ever tried to quantify that brand value or future value?

We haven't. No, we haven't. What we just have is a group of, I think, five what is on the order of five leaders that have a very good, good feeling about, no, this is, this is too much or know this is not good, and I think they're feeling is quite good.

I would say we run into this problem sometimes. Now we do run an experiment. And where we have to make a trade off, like for example, we are in some experiment because we know if we do this, we lose ten thousand users, would make this much more money or whatever. We've never known how to make these these straight, ffs. So generally, what we do is we go for users as in if something decreases the number of active users that we have unless it's gonna make us five hundred million dollars, we're really not going to launch IT.

Just think of laugh c bit. Check using that light on me. This applies to any business like for us if you told bed on me, okay, go. Objective function, maximized the value of required. We could come up in ten minutes with a whole list of things that we know exactly what we were doing that would make .

an obscene amt of welcome to this daily episode of required.

Yes, I would. But for the next few months, and not for the next few.

i'm not even sure that that like that might be the value exists zing thing to do, but we would never do that because I would just kill our hearts, see you like.

yeah that that also matters a lot yeah it's funny though your heart, you can boil frog quite well. And product managers that do lingo have gotten really good at boiling my heart slowly. Things that I think I was not okay with five years ago. I think i'm a lot more okay with because they just kind of push the envelope of every sleepy y slopes they they get there.

But do you end up feeling like, okay, you know what? They had a point like I no longer and sometimes .

i've definitely been wrong about stuff. I mean, that's this for sure. But I still claim in the long term, IT really is Better to do what's best for the user. And I don't mean what the user is asking for because the user for weird IT, I mean, what you think is Better for the user. I think I think that the end, the long time, I am pretty sure that this value maximum.

I think this is a pretty specific thing to founder LED companies, which is we will do what I personally feel in my gut is best for the users and like that just Carrying the day every time. That does not happen in a professional management setting.

IT may not. What i'm happy about is, I would say five years ago, I was solely my gut. And I think at this point there is A A group of leaders. Gemer had a product similar chief design officer that whose god somehow is like identical to mine at this point may one percent different. But I I feel pretty good about the fact that if I were hit by a bus, these people's gud's is pretty so we've .

talked a little bit bit this idea that you a bottom of funnel governor on what goes out, and we've talked about that there's mid funnel thing. That's the A, B test. And I think you ve said in the past.

the latest number is fifty.

fifty. Okay.

this is early, close to fifty. Fifty as in like IT really is fifty. Fifty is not forty eight, fifty two. I can't explain one of those one, just like really fifty, fifty common people has to be something different, but now it's pretty to fifty.

So that gets to this question, why is IT that over time, your collective gut has gotten Better about what experiments will succeed? The way has .

gotten Better is because we get the resource of experiments. Our god is kind of a learning machine where type is at this point. We have years of experience that people have been doing, product review have been doing that for years now.

We have years of experience knowing this type of changed, those that type of thing. And we're pretty good at IT. We're not one hundred percent, but we're pretty good at IT. Not only do you know which direction things are gonna go, but you also know which ones have more impacts of.

For example, we know that the session and screen, so that is the screen that you get after doing a lesson, matters a lot in terms of getting you to come back the next day. But we know that other screens, like screen in the middle of a lesson, almost matters not at all in terms of getting you to come back the next day. So we just kind of learn what what screens matter and what parts of the APP matter for certain metrics. We just have a really good gut on that HMM.

Do you have to treat big experiments differently than little experiments where your gut may not have as much training data in IT?

yeah. And there are some experiments that are a lot more complicated than others. Some stuff takes us a long time.

Well, first of all, now we also have pretty sophisticated da science that tells us for how long to run certain experiments. We also have a prety good god feeling about this particular experiment should run for a pretty long time. Most experiments were run by the we run them for two weeks. We have enough data, two weeks to know where to go. But there are some experiments that we run for about six months because we know that that you're going to see a very big difference at the beginning, then later because people get used to the thing.

There's a very heavy statistics component to the core competency of your company. Where did that come from?

I mean, the first many employees were basically computer nurse from carnegie million. I think that's where I came from. We understood things like statistical significance coming in.

Yeah, makes sense. IT. Does IT feel related to your crowd sourcing research? Or does that feel like you you brought to what on who was a super key higher, and then, you know made that a pillar of the company?

I don't know if it's exactly my cloud sourcing research, but it's generally the first few people that work at the company. We had a pretty good grasp of statistics. What I will tell you though is that it's funny.

This is true for most of the things, and I don't think this is unique to do a linger for most things. You know, we have a pretty good grass of statistics that's good, but the level of sophistication that we do analysis today compared to that is insane. I mean, we now have people with P, H, decent statistics who actually, honest to god, know what they're doing.

I suppose we had a pretty good grasp statistics early on when we were figuring out how to teach things. You know, we had an idea of how to teach things by now. Now we've hired dozens of people with P, H, D.

And second language acquisition and the level of sophistication that they in terms of learning outcomes and stuff like that compared to what we were doing at the beginning is night and day. So we had some expertise for a lot of things, but today we just we have become a lot more sophisticated about most things, including the statistical analysis of far experiments. Knowing what I know today, i'm pretty sure that experiments for eight years ago, many of them we analyze wrong.

What do we know we're like? Yeah, we look at averages, we look at statistic significance. We were not looking at the stuff that people look at today. In fact, I don't even understand what they're doing. So very smart people come and tell me, do this I like, sounds good.

So all this to me, and I bet to many listening, sounds like the perfect T. A. A.

I. You are perfectly suit company to A I, maybe just at the highest level. What's going on? What's changing? How is IT impacting everything we just talked about?

You are right. We are very well suit for for A I. I mean, we have been the whole time, you know A I of course means something a little different today than in ten years ago. Today has been around for a while.

but sce ten years old, I meant everything we were just talking about.

Yes, yes, statistics. You know, since we launched, we were very clear that what we wanted to do is we wanted to have a computer teach a language, not another human tea. Another human can teach her your language very well.

That works, no problem. But having another human teacher language is expensive. We wanted to have a computer teacher language.

So because of that, we are very well suit. We need to make a computer that is as good as a human teacher. And so we've been working on doing that as much as we can over the last couple of years.

What has changed is these large language models. And again, I emphasize that the second l is language. We are a language learning company. Large language models are particularly good with language. And because of that, we've been using them a lot.

I mean, so i'll tell you the two big things where we where we're seeing large language models be used as used to be the case that a lot of the data that we created for a courses was half hand made. So what that I meant, we made tools, but usually a lot of the stuff with humans doing things. And then, you know, maybe, maybe the computers would fix that up or what, but it's kind of half hand made.

We've gotten to the point where most is not a hundred percent, but most of the stuff that that we're creating data for language learning is made by N A. I. There's a couple of good things about the first ball IT saves this money to so in the humans.

But the most interesting thing that has happened is we cannot create data so much faster, and it's not that we can go fast. Okay, that's nice. It's that, that allows you to do things that you weren't going to do before.

So for example, about five years ago, somebody pitched a feature on the single, which was a type of lesson where is kind of like a mini quit, like a two minute podcast, where you basically learn how to listen in the language that your learning is like, looks like a little animate thing. We call IT do a radio now, but I was like a thing where you could watch two minutes of something and listen to IT, and you're learning how to listen. And I was told that creating the data was going to take five years.

And as soon as I was told that, I was like, no, no, this feature red light here do not proceed. I do not want anybody working on five years of seeing something that only even know it's gonna a very similar feature. Now we can create the data for this in couple months. And so now i'm like, yeah, why not what is IT about today .

that caused all data done by a computer are .

supposed to having done by hand for IT to make a difference in the dulin le language learning, we need to make hours of content, many, many hours of content. And specifically.

in this case, you're talking about like animated graphics.

not just animated graphics, the audio of IT like actually, you know what you're going to say, you have to make IT interesting. So you had to make something that people had to listen that was interesting and said that you had to do all that is super slow to make hours of content. And I was going to take five years to make, and not you to, in all the languages that we teach, we teach forty languages, and we had to make hours of content.

So that was going to take an an ordinary amount time. But now, you know, with an, you can do IT pretty fast. And what we do is we do IT, then we kind of spot check IT.

And if we don't like IT, do IT again, we're very happy with that, that you can basically do things that before with just they were not impossible to do, but practically impossible to like why would you do that? That's an important place where we you say I the other big one is just conversational practice. We are working on this feature that some of our users already have. We haven't fully really launched to everybody, but IT is basically but you can have a conversation with one of the dulko animate characters and and IT looks like you're having a video call with an animated character and is a really good way to practice a language. Actual conversation, that's something we just couldn't do before.

Yeah i've heard mark archer g say about lama use cases a few times or maybe it's about meet A I use cases that it's for people to practice conversation that they don't want to have yet in real life, but they need to sort of feel like they're having the conversation and get some authentic responses.

Well, so that in the case of language learning, what's amazing about IT is, look, you wants to learn language, have to be able to practice conversation. You just have to do that and people don't want to do that. In fact, we've done insane amounts of user studies where people they may tell you they want to practice language person, but when IT comes down to IT, they don't want to do IT.

There's some small fraction of extreme experts who like IT, but the majority of people would just rather not talk to another person in a language that they're not very good at. But the beautiful thing here is that you can just talk to an animal character and you really don't care because you don't think they're judging you even though technically they're really judging you. That's mainly what the thing is doing. But do you just kind of them .

think about that? That's just absolutely incredible. And A I data also opens up the other direction for you to like you have an incredible amount of data within duo lingo that can be used and feed back into LLM your own, I would assure, and others, that would just be incredibly, incredibly valuable. Does that open up a whole another new business dimension for you .

all maybe at the moment? Does not something we're doing. We're using the data for ourselves.

More than one billion exercises are solved by our users single day. So we have the data and basically, we are watching. I know tens of millions of people learn a language. Every single. We're using IT to teach Better.

Today, we know the dual language about as good as a classroom in terms of learning a language, but not as not yet as good as a good human tutor like a one and one good human touter. The promise, those those are like fifty dollars an hour like a good one. We're not as good as that.

yeah. It's hard to break the future. But i'm assuming ing that if you give us three years, will be will be about as good as a human duty well.

And IT requires making an hour. I mean, if you're doing three minutes at a time, that's a structural .

advantage where match back.

yes, you're just able to address people that will never ever make the hour.

yes. And and you can have a paid human touter that is there for three minutes and then another three minutes later .

and then they would have to be sitting in the .

style next year just waiting around.

It's how do you say three years? I wonder if it's fast because like you got two tracks of progress happening that are accelerating. You've got the foundation models of LLM themselves, what you're getting Better and then you've got your data, which you're getting, what did you say?

Ten million solve .

exercises a day, not a day. So the quality of both of those are accelerating. So i'm really .

bad at predicting the future, how long things are gonna what .

humans are and you're a humans.

So yeah, I just is very hard at it's a matter of a small number of years to get as good as a human .

to you're in this education space. Are there any education companies that you look up to as shining examples of people who build great companies? Or do you mostly look outside your sector for inspiration?

It's mostly outside education has been hard because you wants to start an education company. You're learn to a lot of problems. First of for some reason, most education companies have decided to monetize by charging the school systems, which is fine.

But if we just can't look up to those companies because we're so different, we are consumer company. We've having the past look at those but is very little to learn from somebody who has spent a lot of time making a sales force for cafe twelve schools. I'm glad they're doing IT good for them.

IT may work, I don't know, but it's just not much first to learn. So ultimately, for us where we learn because we are a direct consumer, we learn more from games companies like spotify, netflix, any of the matter apps. That's where we learn the most, but is because we are direct consumer. There's just not very many direct consumer scaled education companies.

Well, there are, but they are all non profits with brands like to C M U or .

harvard for yes, yes, they are there, but they are not apps and they are not touching a hundred million people, like touching a few thousand people. But yes.

it's kind interesting to think about, I mean, they started, as every institution starts, as this sort of niche thing, using the latest and greatest st technology available. Then over time, you build a durable brand and you try not to jeopardize that brand in any way. And those words that I just said could apply to do a lingo and you're just like seventy five years earlier on your journey than that is. Other institutions like, like is the goal to become a accreditation bearing institution where people feel like when you've graduated from that means something I would really like that we .

are doing that for a language we try to do for a language. We have a standardized english that we don't have for english, mainly because that's where IT matters of this test called the dole in the english test. And the idea is that this is a test that you can go and take IT.

You can take IT online, and IT gives you a score, and IT tells you how good ear english ous. And that is actually valuable, for example, is now accepted by most universities in the U. S.

If you are a foreign student that wants to come to the U. S, you have to take an english test. Ninety eight out of the top one hundred U.

S. Universities, according to us, us and world report, accept our test. We have been quite connected the test and the APP, but we want to, for some languages are already there.

We're giving you estimated test score. So if you just use the APP, we will be like, oh, my test score is this much and then you can go and certify the score. Know, what I would love to do is to be able to own the score for language.

So right now, if you ask somebody, how much, how much french do you know? The answer is usually something to the effect of, I took four years of high school, french or intermediate, but that doesn't really tell you what we would love to get to. The point is, if when you ask somebody how much differently know, they'll say, oh, a do lingle sixty five.

We're not there yet right now. People tell they're using dillinger and they may tell you they're dillinger streak, but that's something that we want to get to. And I think that will be really valuable. You're right. I mean, maybe in seventy five years, we'll just have a really stodgy stone buildings and a thirty billion dollar in downtown.

Well, it's what you took. I would like what would the market cap of stanford be if I were you publicly to corporation like IT would be quite large.

IT would. And you know we have this board member being gordon who who who's amazing and he'll happily tell you he'll say market cap of a universe twice its endowment that feels .

like a low multiple to me. I don't know .

where he pulled that from, but this is why he'll tell you, I don't know, I don't know. He pull, everybody will tell you. So stamford, probably, you know, I don't know what they are down in.

is thirty billion forty?

yeah. okay. So eighty billion dog company.

This is such an interesting that experiment, because the mission of education is so unbelievably important. But IT has history ally in a twenty thirty years, been so hard to invest in education technology companies because they so rarely are using modern business models, the technology that all the other consumer companies are using, the distribution methods that all the other consumer companies are using. They are instead sort of trying to snap to, how do I sell into the education supply chain and you just have IT like you might be the one example of this education technology company. That said, I am not playing in that existing system, and that is how you invent something new IT vars on the day.

But we're certain ly the most valuable at the company. We may be the most valuable education company. Some day's person is more valuable than some day's were more valuable than them. But and what's interesting is we doing that with just one APP. And if you look at something like pears and they sell gazillion an books and stuff like that, I mean, we feel pretty good about that, that we really are actually able to teach Better. But you know, we feel pretty good about that.

right? I want to a close with a question on heddon dsg tailwind. What's the most insane tailwind that just makes things easy for you running? Do a lingo and then other side of what is a headwind that you think you are just always gonna to deal with and it's a throwing in your side. But that's alright.

A I certainly a big tail when for us right now, I think there is a growing proportion of people who want to do useful things online. And I think we get tail when from that. I think if you use two little for thirty minutes, you don't feel the same as if you use instagram for thirty minutes where you're like, my god, just last thirty minutes of my life.

I do think that's a real tail win for us in terms of headwinds. This is more for me as a person. Employee problems are tough.

You know, we hire the top of the talent pool. I'll use the word enticement. Is there a stuff for me? I grew up in a country where, literally, for two, three hours a day, there was no electricity. There are some people here who are like, why do we only have two flavors of coconut water? That's always hard.

It's funny to connect that with something you are saying earlier. Like part of why that exists is the X, Y, Z, A lead educational institutions in these days in their brands and their brain caption. Like if they were just to do a lingo score of like how good a product manager you are that with level, the playing feel Better.

Yeah exactly. We could get this people that weren't to fed with a silver spoon.

As much to go back to your comment of you grew up in an area where you know there were multiple hours of the day area of electricity. How much do you feel like growing up in guadala? I think you went to one of the sort of rare private english speaking schools to have the opportunity to learn english. How much of that formative experience for you contributed to you found into a lego?

It's a lot. I mean, first of, in a plastic watermill language, learning is life changing. That was a big deal.

But I think the other thing is, I was fortunate. I received the rich persons education. Even though I didn't grow rich.

I was fortunate that my mother spent basically all her money on sending me to the fancy school. So I went to school with very rich kids. It's not like we were poor.

We were not, but we were your kind of middle school. And auto moa, which probably for us standards is poor. But in auto mo, we were middle ass.

I really saw the difference because where I lived in the neighbourhood, I lived the kids, we didn't go to the school that I went to, and I could see how much more I was learning. I think I was pretty clear that I was learning. I know twice as much as they were.

That was a big thing when when starting doing so, we wanted to do something that would give no access to education to everybody. So was a big thing, me growing up. And what am I had a lot to do .

with starting to linger. Well, that's a great place to leave IT. Louis.

thank you so much for journey us then, David, thank you. Great questions.

Unless there's we'll see the next time.

We'll see the next time.