Building Your Own AI Agent with 11 Labs
10:02 Share

Welcome to the A I chat podcasting. On the podcast we're talking about, one of my favorite A I companies, which is eleven labs, are new thing. They just launched ed, which is essentially the ability that they are now offering for you to build conversational AI agency for the leveraging their voice models.

And they're actually letting you build agents rate on their platform that people can talk to VISA conversational customer support. These are great for um you know being a travel agent or helping you take an order at a restaurant. There's so ways that these are going to be used. I'm super interested. I'm going to be breaking down.

I'm actually kind of walking through some other documentation for their developers on how to set up an agent, and i'm just going to essentially do that to explain to you how you build one of these things I try to do in the simple so if possible, and some of the capabilities that that helps. I think this is really interesting and this is kind of a bit of A A deeper dive on um how this actually works. I think it's important because this is going to give you a really good vision where this goes in the future, how this works and kind of everything you need to know.

I want that comes to that. So overall, I very exciting topic. And before begin to that, I wanted to say if if you wanted get a daily newsletter, sharing daily AI tips and different stories and tools that are breaking, go check out A I box or AI at the bottom of the page.

There is a sign up if you put in your email and i'll send you every single day the top three AI news stories and just a little blair began super long. I'm trying to make short, useful, concise. A lot of people use that with thousands people subscribed.

And so if you're interested, you go get that newsletter and I will share daily kind of what's going on in the AI environment in the early hands on way will see you can see tweet what people are saying about the different stories is really interesting and try to make the thing useful for you. So if you're interested to check that out, there's linking the description to A I box. stop.

Ai, okay. So what exactly the eleven labs building here? How does this whole thing work? Um I think up until this point, the eleven labs is not known mostly just for giving about different AI voices. They do text to speech.

Their head of growth, sam skylar, was talking to tech cranch I was hole thing and said that a lot of the clients were already using the ability of the new conversation al AI agent um to to build some stuff. Um so in all of this they are able to use what the language this agent is a is speaking, what the first message is, what the system prompt is right, to determine kind of what the agents persons is right. So you will be like, hey, act like a traveler and always say this, always do that right?

So you can can put all of that in and then you get to select what L M. You're using gi GPT or caught, which is pretty awesome. And they got to a different like temperature responses and and different settings can change.

So it's really kind of design to be hands on. So I was over looking at looking at eleven thousand and how you actually set this up. They have a really interesting demo of kind of this whole tool in action where they have a travel agent that we've built and the travel agent essentially is talking to someone about a trip, they're asking adventure, different questions.

And I don't want this. I want that. And what was really interesting to me is at the end of the conversation that I had, which I did great, if you know, eleven lamps or audio sounds like pretty decent as one of the best, I think a as far as voice goes out there.

And of course, opening, I has a great voice. But IT doesn't nowhere near as many integrations and kind of suffix can do. So at the end of the conversation, IT then has the history, which I break down is pretty cool.

So in their um in their kind of documentation, they explain how to build one of these tools. They are building one for a fake store called perogue palace if that means if you've ever had perogues before, they are delicious. There are polish thing is you know get like math potto es and cheese in the middle of like kind like a dumpling maybe I don't know, in a little hot pocket kind of thing anyways there delicious big them anyway.

Um so the thing that you're going to be able to do first with your assistant set up is that you you go to the dashboard, um you cook on, create assistant. You can either do IT with a template or you can choose a blank temper, which is kind of interesting, right? So they have one that is like a support agent.

They have a temper for that, which makes IT really easy for you. They have won as a video game character. They have won that's a math tuder. And then you just do a like temple's. So depending on what you're interested in, you got a couple options to get the thing started.

Then you're going to go inside your first message or your system prompt, which is essentially, you know, you could say something like welcome to paroe palace m here to help you place your order. What can I get started for you today? right? So that's like the greedy message that always says then you do what is called the system prompt. And the system prompt is essentially you telling me exactly how how to act.

You tell me what kinds of things to say in their example that they gave this, that you're A A friendly virtual assistant for a probe palace um a polish restaurant specializing approaches is located in the zacatecas ounce ins and poland so you really get specific and I mean like I can imagine how to restaurants will use this exact tool they said your rolls to help customers place order over voice conversations. You have comprehensive knowledge and menu items on their plate Prices. So what's interesting is you can essentially use this.

You can build kind of your agent or whatever your conversation, al agent, and you can tie this to your phone lines when people call your restaurant. IT could be greeted by this. You could even have two right? One that's like screens, calls like, hey, what can I help you with today? And then depending on what you want, it'll send you over to like older, trying to make an order.

Okay, send you over here all the time. Mic reservation. Maybe I can take IT or senate to another one that set up for reservations that's tied into our calendar. There's a lot of cool things you can do with this. Um I really excited in their special system prompt, they actually listed out with the menu items works to like potatoes and cheese pogue three polish slowly per dozen that like Prices and items which I think is really good.

You used to copy pace a whole menu on here and it's only pulling from this such as from like, hey, I want to order this random beef stroking off and it's like, okay, sounds great, right? IT just looks at what's on this and only lets them do what on IT reis rape so and then also uh in the in the process that they showed like their example, they said these are your tasks, create the customer and kind of says, I do that. Take the order and says, listen carefully.

The selection blob, confirm the order, calculate the total Price, collect delivery information, estimate delivery time, provide order or summary, close the conversation so as an exact flow of how this conversation works. And then that says guidelines, use a friendly, professional tone through out the conversation, be patient and attentive to the customer's needs. If you're not sure as console to repeat, do not collect any payment information, just tell them the payment will be handled upon delivery, avoid discussing topics unrelated to taking. Imagine the order.

right? They're doing that because so I could be like and what's your opinion on the current political state of log? You can get these things off the rails. So they're really have to put up a message in there to to stick to the scrip, which is kind of funny.

Okay, once you have that in place, once you have both your greedy message, so the first message and then also the system prompt in place, you then go and you can actually can figure your voice settings so you can choose from over three thousand different voices that they currently have eleven labs um and yeah, you can listen them, you can test them. This bunch they have a whole marketplace where people upload their own voices. So there's a ton that's in there that's really cool.

Anything they comprised people. So it's it's an interesting thing. So then after you do that, you go and actually test your assistance. So you to have a little example button where you press more and then you have a whole conversation with that.

The system after that happens, you're need to configure how the data collection is handled so you can configure how um you collect and analyze all of the conversations to like essentially you can go and look through what was said. The transcript IT also makes like a little AI summary, which is kind of cool of the conversation. Um they have an analysis section for the city for the assistance settings where you can essentially define custom criteria.

Ia if you're trying to like evaluate specific things in a conversation. Um and yet there's a lot of cool tools they have there, one of them as a go prompt criteria. So this passes a conversation transcript een l to verify if the specific goals was met, right?

So you could say the goal is to give them information about the menu or the goal is to have them buy something and then you can run IT kind of through an L M. It's like based off this conversation, was this a success, failure or unknown? So they have a bunch really cool tools like that.

Um you can set up how you want to collect alive the data, including um you know the customer was a name and all that kind of stuff. And then you can go view the entire history of the conversation so uh, you can see kind of a summary of IT and then you can see everything that was said in the conversation, which is really cool. There's a lot of really cool stuff that we've had built into this.

So after that, you're ready to go and your tools ready to take orders. Overall, I think this is a fascinating time. Eleven lives is really pushing forward as one of the key players, their competing obviously against opening ice whisper assembly, a AI deep gram speech matic.

Yeah a lot of others. And they're also trying to raise right now evaluation of three billion dollars. So they're actively trying to raise money. I personally think that eleven labs is. One of the best A I companies out there, one of the best audio A I companies out there.

So i'm a huge fan of eleven labs, been using them since the beginning and have seen that they are only getting Better and Better three billion dollar valuation, honestly, to me, sounds perfectly reasonable. They compete directly with OpenAI while opening eye has demos, some cool voice tools and and stuff. Eleven lab just beats them to the punches as far as getting out the door.

So I don't know what eleven labs is cooking up right now to compete with that, but a lot of those tools aren't know. All of those tools are not even out from opening eyes. So I think there's a lot, a lot going on there that they have a lot of opportunity.

In any case, if you enjoyed the podcast today, make sure to leave a of you wherever you get your podcast, make sure to go and sign up for the newsletter. If you're interest in getting daily um E A I news into inbox, hopefully a really useful way to you. Thanks so much for tuning into the podcast day, really appreciated. Hope that you all have a fantastic day, and I will catch you next time.