Graphs and ML for Robotics
41:59 Share

You're listening to data skeptic graphs and networks, the podcast expLoring how the graph data structure has an impact in science, industry and elsewhere. We're welcome to another installation of data, a skeptic graphs and networks. So they we're talking about the interesting problem of room classification. So given a flor plan, can you determine what's a bathroom, what's a bedroom and so on so forth, specifically doing that with graph neural networks. Assad, what are your thoughts on this .

implementation? What I like about today's guests is that he shows us how to use graphs to add more relevant features to machine learn. The pretty thing about graphs is that without adding more data, just by utilizing the data we have, we can add more dimensions to our data set.

By utilizing, I need model your data as a graph, and so you can use properties as features, for example, as we will soon hear in the episode, if we take rooms and bundle them as notes and then model, let's say, the doors as edges. Patterns in the graphs we've made could help us to Better classify the different rooms, but even cooler is because it's a graph. We can look for patterns not only in the immediate surrounding of the node or the room, in our case, meaning its sneider bors, but also he knows that are two or three hops away from IT.

Elfish c refers to IT as the topology adaptive graph. Convolution in everything that aisha did was to create a ways like APP, the navigating APP for a robot school navigating apartment. So like way is the robots use algorithms like justa and a star to find the most efficient way to get to where they are supposed to go.

But since it's a graph, he can also add tears to the data, making the rooms that say the first year, then the furniture of the second and the items on the furniture of the third. So you can nap everything in the house to the level of the tooth space sitting beside the sink in the bathroom. Each of them can be anode by itself.

Personally, the part that I really like the most into conversation IT was about how our guest makes robots reflect on their mistakes. Yes, I can check off the the image in in my mind of a distressed robot called crying to itself. What did I do wrong? What did I do wrong? Said to do this, that soul searching in robots sound really just.

yeah, very need aspect of machine learning that is somewhat knew. I don't know this is the first use case or whatever, but know classic reinforcement learning is very pavlovian. You ve got minus one for doing the wrong thing, and you ve got plus one for doing the right thing.

But here it's more to say. You made an error during this epic of time. Go reflect on your error and, you know, even the potential to simulate other circumstances on one zone infrastructure, and then figure out I had to build a Better algorithm or policy for the next one. Really interesting learning style.

We evolved from the behavior. And even in robots, but even in robots, IT doesn't work as as much as we wanted to. We need something more, more elaborate, right? It's not so it's not so simple. So it's it's cool to see that with the phenomenon we see in the humans, we also see in robots.

And to your point about how they extract topological features from this graph, that is the most common news case in industry. I observe where people are involved in graph s. It's that they extract some features from graphs, not just nearest neighbors, but second, third level neighbors or structures, to the graph, a company triangle als or bothy structures you have.

And then those just become new features in traditional machine learning, although different from what alby shack us today, where they're actually using a graph neural network that sort of inherently takes the structure into account, which is a nice feature. But I will have to get more into the inner ds of graph ual networks in the future. And true, definitely want to pick your brain and your thoughts there. Let's team right in the interview then.

So on a computer science, A P S, D student at George mission university in service ia. My rest interest in areas eros machine learning and planning and uncertainty. And my research has amy. I'm more of practically interested in like how to into a robot behavior and abilities uh to adapt in multiple environments when the robot es like using some kind of a machine learning and things. So yeah moly on that area for for improving robot learning.

So I did take one robotics course in graduate school and for some time ago and I was kind of surprised at how uh that a simulation based IT was. We didn't talk about fiscal robots. We are talking about fast lam and things like that all in software.

And I was surprised to learn IT felt very disconnected from the physical side. You have the same impression. Or perhaps is that changed over time?

That has changed over time. I developed algorithms that are first demonstrated to work and simulation and then we are sure that, look, this works in simulation and then we try to like put that into a real hardware robot or something um until at this point, like I have a lot of algorithms that I have demonstrated to work in simulation and then I am working on like transporting like those things into the real robot so that we can demonstrate that this also works in real robot.

So I think there's definitely like a baLance to be hard, right? You can like do a lot of things in simulation, but until you face the real world, you cannot really see IT that. Okay, this gonna work, right? So a lot of the folks that do like label most and control, I think they are more focused, heavily focus to on demonstrating the cap of that is in real hardware, right?

Because that's the of their work, unless you can like the move that toward can handle low table controls in the real robot itself than just doing in IT in simulation might not be as effective way to demonstrate their work. All the folks that would who would like to a highly bal planning, like for example, where the rose will go, whether they will interact bathroom or into a km if he is trying to find some object. Those kind of like highball planning can be demonstrated even without the need for a real report.

I could demonstrate that, okay, or what enters bathroom to find less? They are toothed. Instead of going to the kitchen rights, you can demonstrate that even without the rival robot moving from maybe like a reading room to bathroom. So this definitely like of the focus of what kind of work you are doing, that pride I is, whether you actually demonstrate in real, really about or not.

So and how has the topic of deep learning and deep neural networks impacted the field?

Oh yeah, I mean that that everyone everyone's doing that, right? So I think global s is not away from from the from the boom of A I and deep learning, right. IT has actually like brought a lot of progress in when deep learning has advances, especially with the computer vision boom starting from around like to twill with the image ate.

So a lot of the vision based understanding has improved that has allowed, like the perception system of the robust to improve and improvement perception 呃 systems of the robot has actually resulted in lot of improvement in robot behavior。 right? Can you can identify objects Better? So I can once you identify objects Better, IT can go and find that object more efficiently.

So things like that, right? And the recent boom in NLP, like people are trying a lot of things with alan and robots. There's a lot of interest recently, but I still I think I think that there's a lot of work to be done in that area with with the new deep planning tools like ala ams right now.

Yeah and is is really exciting. It's in time to be in robotics. You know like a lot of the research with Adams are being done and then you have the Operative ity to like get those kind of the new exciting ideas into robotics to improve, improve how the robot behaves in the real world.

Well, I recently read your paper room classification on floor plan graphs using graph neural networks. For listers who haven't taken a look yet, could you give a quick summary of the effort?

IT IT seems like a little deviation from what I talk about, robotics and all. But I think this was like an old paper of mine. I think are those are like, I don't twenty to anyone.

So that was when I was just beginning to work on robotics. I was working on like I was, I was basically first time getting started at doing the bodies. And then there was a lot of ideas about how to map an environment right for the robot, map the environment with the lighter sensors.

And also there is a lot of studying that I was learning about a robot mapping. And when the robot maps and environ, right, this baby, like building a two degree right with the obstacles are and where the free space are. And a lot of this idea got me interested about what? How about, like, robot? Whenever is, like, building a map in some building?

When is, like, maybe a household or an apartment building, right? And then I ask you to, like, bring me a tool pace. Like from an earlier example, IT would have to know where IT is fresh.

Where IT is situated right now is a living room or a kitchen or a bathroom. And once IT identifies that, you would have to kind of reason about where the toothed might be, right? Maybe IT might be in the bathroom. But so IT has to look for where the path room is.

And wherever he goes, IT has to identify where the, what, what room IT is currently in, right? So this kind of fact, the motivation that got me into into this idea room classification with four times right, the nobody is given a map and IT has to identify what rooms are in the map first before like going into finding like whether to this might be right. So that's that's kind of like the seat idea that got me interested into and this research of classifying rooms in a map based on like whatever data that you have about the about the environment, that's basically like how the idea what is an .

underlying data set you can have to ask questions like this.

This episode dis brought to you by work OS. If you're building A B to b sas APP, at some point your customers will start asking for enterprise features like single signa skin provisions, all based access control and audit trails. That's where work O S comes in with easy use and flexible aps that help you ship enterprise features on day one without slowing down for your court product development.

Today, some of the hot startups in the world are already powered by work O S, including ones you probably know like perplexity, versie, jasper and web flow worker. West also provides a generous free tier after one million monthly active users for its user management solution, making IT a perfect authentication and authorization solution for growing companies. IT come standard with rich features like social loggins, bad protection, M, F, A roles and permissions and more. If you're currently looking to build as a soul for your first enterprise customer, you should consider using work O S in a great minutes and start shipping enterprise plans today. Check IT out at work O S dot come, that work O S not come.

Well, what is an underlying data set you can tap to ask questions like this?

We looked into into the data is for the floor planes is specifically because with floor plans, what you have is basically the structure of the map, like the top view of the man uh we liked into a into the data um about like what we could find on the internet and week came across this dataset from a houses in japan, so houses and apartments in japan.

So I think it's called I have to check, I think it's called leader house data, da said, or something. So this can basically contains uh, flour plants from, I think, around like one hundred and forty three thousand houses and apartments in japan. And then they were like levels about what the rooms were, what the rooms coordinate were, right? That's where we find we found a tera tera for like doing this this experiment that we we thought about.

One, curious if you could expand on the data set. I'm imagining IT could be something like image net except pictures of floor plans. Although that sounds especially unmanageable. I'd rather have like a cat file or something like that. What is the feature of the data?

So the lethal dataset, house dataset, I think, is the origin dataset that we came across and we also came across. I think this was like a very raw flow time that I said. I think IT was like the rater images.

Even we did not directly work with the legal directed because we found another work that someone has already someone had already done that had converted this rasta flow plan into like a Victorious format and then extracted this coordinate of the rooms from this floor plan. So a lot of the work that was made easier for us by, uh, I think the paper was, how can. So what they did was completely difference.

So they were doing something with this idea of generating flow plans. So basically generated. But for houses and in the floor pantry. So what they did was basically they took this, uh, lethal data set and then extracted vectors from that, and then have the room coronets and everything extracted from this, the images, I think, and then they use that dataset to learn a generated adversary, newer network to come up with new flower plants.

We kind of like borrowed a lot of the world from that house can paper that converted this, uh, rest images to Victorious, from and with coordinate. And also a lot of the work we didn't have to do someone else needed for hospice. And that's what are a lot of the researchers already. You don't have to have to start everything from the sats. You can build on top of other people's work.

And that's what we did like with the data processing part of this uh of this word for room classification so basically this day, as I had like room bounding boxes, like and then room a room categories, the coldness of the was ways of these rooms which while corresponded to which rooms and then whether there was two years on the wars and so so so so on, right? And there was also like rv images, the actual images that was used to enjoy the full transport. We didn't really use that because we is needed the veterans form of the data. So ah well.

having that house san data set sounds like a great starting point, yes, but uh, it's still not clear how you go from coordinate and poly guns to sing a graph is the right tool here, right? What that inspiration around. So in robotics .

is this idea. So we talked about occupant's map map representation with as a two degree, where each great sale could either be free or an obstacle, right? And and when you have a large create and E, L is either free or obstacle than you can like represent a big uh floor as an occupancy great map.

This, this, this other representation of map, what we call like topological map, I think yeah topological map. So the representation would basically have prominent landMarks. For example, a landmark could be the sap point in a living room, right? And then you have you have a path.

You have like a straight line connecting, uh, that point in living room with maybe the door. And then the robot can move from that point in living room to the door. And then the door would be represent as a note in the living room.

A point in the living room would also be represented as node. And if the robot can move from, likely, say, the front tour to a living room is an is that connects these two notes. So these topological maps are used in like robot planning, right?

So if the robots at the door and IT wants to go to bedroom and that there's like a note from tour to living room and then living room to the bedroom, then you can plan a path from tour through the bedroom, right? So through the living room. So this, so this idea top los can map is what or what kind of like inspired us to represent this room and the whole fall plans as a graph.

So should I expect one node per room? Or would there be many nodes that represent various positions inside one room?

In this research, actually, we represented one room as, like one node in the graph, something like this, right? If they are like five rooms less, say, right in this, in one floor plan would have five nodes. I think the connection between the ages, I have to refer to the paper again, because is from twenty twenty ones.

I don't, I want to, I, I don't want to be wrong about this, right? So I think we connected the two nodes in this room, nodes graph, if they are acting within within, like some distance thresh hold three percent. So if if like the rooms, if the two notes are winning, like the three percent freehold compared to, like the whole four band land, then we connect these two rooms with an eyes. And that's how we construct the graph from these four plants.

And then is the edge just the representation that I can move from one room to another?

Not exactly. For example, sometimes a bathroom, or like this, a living room can be separated by one war with less the a dining room, but there might not be pass from the living room to the attaching room. Maybe you have to go to kitchen for us or something like that, right? But if they have like a lot of thing, while or something in our method, we still connect that living room with a tiny room.

The connection, the ages connection, are based only on how far these rooms are, not whether you can directly go from that room to another room. As I said, I mean, in robotics, like in in the topological maps we talk about, that was the inspiration, but we did not exactly represent this floor plans as a topological map where you can navigate to the north to the ages from no to know, right? So there there's some certain differences. Here was the inspiration, but we thought this was like a Better representation then because we are not really like interested much like, I mean, that could be a different focus where where bring the like leveraging that people will map itself in this research, but we wanted to like make IT more viable for for what we can learn for draft neal networks.

We just start of curiosity though, I have to say I can't remember a time when I ever was in a home where you could go from the kitchen directly into the bathroom.

right?

That's a structural insight that maybe you could take advantage. You have any thoughts on, you know, maybe not having that in your model?

I mean, after we did this work on them like a few months after we could see a lot of a improvement that we could do, end the thing that you mention like actually like representing as is to be like what they they can traverse betwen nose is is a really, really good insight. But we didn't really work on that because I think I think we had like some kind of tiny meat.

I was doing this as a also as a part of a course project that I had to tell me one time. So there is a lot of limitations about what we could explore and what we could not. And we kind of like do not export that side of of the things because we had like some time question before we could finish this research.

But definitely like if we have more time, we are even after like we submit, we kind of like thought about that. But I got more interested into robot planning and then like planning under uncertain, that kind of like this work was left behind in that thing. So but I agree with you that there are a lot of improvements that can do on the plan to post that we have.

I'm curious if you could frame, uh, a little bit more on the machine learning problem. If you think of textbook machine learning, if you get some tabular data where one column is the objective and the other columns of your features, how do you get the final representation going here?

Once we have this graph, we need to like formula this as a classification problem. No classification problem, as we call IT in in the gravity with his nose. We have that represent rooms.

What we are trying to output by inputting this graphic structure into some washington aluisi is a class for ease of the notes that we have in the graph. One note could be classified as a bedroom. The other note could be classified as a kitchen.

We need to, like, have some features associated with each of the north that we have in the grave, right? And those features are water passed into a mussing learning algorithm like a neural network for each room. We had information about the room coordinates, right? what? What's the room coordinates? I think we have six features for each of the room notes.

The first one is every afternoon. So once you have the ordinary of the rooms extracted from the day that we can compute area of the room, maybe generally living rooms might have generally large er area, then they say a bathroom, right? So that kind of like an important feature to distinguish between the room.

So we thought like area would be like a good feature. So we computed the area best on the corner and then put that as the first feature later. At the other feature could be length and wait like previous ly, right?

So the area, the length of the room, the week of the room, how many doors the a room has, maybe a living room has multiple door is right, because is maybe could one entry entrance to her and one that might lead to a kitchen and the other one that might lead to a bedroom, right? The number of doors in a room could also be a distinguish, uh, between room. So the other two interest in features we have is whether the room is parent room or a child room.

There were like rooms law as closet. Many, many light payta have like a working close that are kind of like big. One of the distinguishing feature of closet is. Is generally inside a bedroom. We came up with this, a new fisa that defines whether the size room or not, and that close IT. That inside the bedroom can have a feature, that is, is a child room, and then bedroom can be a part room like converse, right? So those are the features that we use training, the nothing noting out and that we have.

And can you expand on what those algorithms were.

right? The basic neural network algorithms. So we only experimented with neutral network, but we experimented with specifically, specifically with graphical talks, right? Because the notes represent rooms that is is represent whether they are closed or not with graft. Neal network is designed to capture the relationship between nodes, have Better results than regular newer networks like a multiple perceptions on. We kind of like have this multilateral tron as the baseline that, that we were, are comparing with our approach that liver is graph neal network of multiple, a positive one.

Basically, if IT does not take into account this graph is structure that we build, going to take the note features, but there's no interaction built into multiple cept rn and other three we experimented with, well, like different violence of the graph, new netters. The first one is the graph on volume, I think, was one of the first. And this was also inspired from idea, idea convolution newer networks.

And that's why it's got left convolution right IT can eat like a generalized form of convolution tworog. What do you have in convolution N N use like there's one pixel is kind of getting information from the surrounding pixels during this convolution Operations like the newborn pixel, then aggregating that into like a new new feature. The idea is kind of like similar with graph convolution right here in convolution.

You can you can be like consider that a pixel has eight neighbor pixels taking that idea and is like journalizing IT over graph. You have a note and then it's surrounded by its artisan notes and that you are doing convolution similar convolution like Operation over that set of north. So is taking information from the surrounding nose to computer feature in bedding.

And that's how is able to capture the information from its neighbor's to come up with feature. And then we use that feature for, like further transitional ism. So that's like a very high liver of view of brass convolution, right? And that attention network is another variant of the crafts new life that we use IT, kind of like bills. On top of this idea of craft convolution, sometimes what we have is maybe like you want to prioritize one neighbor more than the other nights, or right, the attention mechanism kind of does that for for you.

The other one is graph says so is is kind of like also like similar to graph convolution, but is doing what is what we call like sampling and aggregation, right? That's what is stands for that's what says he stand for in draft right aggregating from all the all the neighbors notes to get ca note features but with grass says you might not be like aggregating from all of the nodes, but like sampling a few few neighbors nodes and then aggregating over that. So aggregation could be like me Operation, like the max Operation like in grass convolution you are always probably like doing me, but grah says like you would like have multiple violence of aggregation Operation, a main million like whatever whatever you have we sample, which no we taken to account for, because we also like use another van called topological adaptive graph convolution network.

So with with all the three graph convolution that we talked about to compete and ebel ding, what we have to do is, like we taken to account what the note embedding for the neighbouring nodes are, right? With one convolution al Operation, we only taken to account the immediate neighbours, right one half, neither our right. So what what a topology active draft from pollution deed was like?

Even with one step, you would like get information, like to complete the note meetings for that in one step, gate information, not just from its immediate neighbours, but like maybe two hot or three hot neighbors. So that kind of like in one step you could get a lot of information from from far away notes. So that's that's kind of like what a technical active gf convolution would do as we can see, like there's a variety here, right? So we wanted to experiment with not just like one kind of variety of graft new dogs. We want to like, explore multiple varieties of graft, relax, so that we could see, like what kind of algorithm, new little graph, new reliable algorithms would perform pair in in this a room classifies and tasks that we have.

Good question for you about features before we jump back into algorithms. Uh, obviously this is a room classification problem. So you the objective is what room type of room is this.

You don't give up the feature but do you tell IT the neighbors? Um like you know it's connected to a bathroom. It's connected .

to a hallway in the feature for a lot. There's there's no hint of what other what the label s for the nose, uh, other neighbor nodes where, right? That is what the meat of graph neural network is. You don't have to like interpret what neighbors are there in the features itself. That information is Carried by the ages of the draft and draft newly talk is in taking into account now is the note features, but also is is to computer and ebing, right? So that feature, even if you don't explicit, have like what neighbors are uh for a note as a note feature graph neuk is able to laborious that implicity through this uh this best mechanism that takes into account of what other notes these notices connected to via cases， right? So that's that's like the billions of the newly talk is what I would say that you don't need to like take into account into the features space what the neighbors are.

So let's maybe start by talking about your baseline. You trained to multiple percept, ron, which lacks graphical structure information. So h hopefully your other approaches did a Better job. But let's started at this baseline. Ah how did you perform?

IT performed okay, I would say right not not really grip. I think we had like uh the test accuracy of uh for for monitary perception, we had test accuracy of fifty five percent compared to Z C M. So surprising.

So is surprising. So Z C M was not able to perform as well as the material. Rn, this was like the first graph, convolution and newer network that was, that was proposed. One, one thing, I should notice that this was like the best performances that we picked from all the experiments that we had, right?

So multiple a process from the best performance over like a many hyper pama tune that we did was five percent and IT could not beat in the craft convolution and new verda which had like accurate acy uh on tested of I think fifty four fifty fight something like year fifty four. But then when you when you look at like other neural networks, right, that that kind of like built on this very vania a graph neal network, they kind of like have Better performance. For example, a grass says has eighty percent and then the topology over has uh everyone percent, which is like higher than than the mlp and other other regular graph in those. So those are like so the baseline mlp is like all performed by the graph is and the total of graph convolution in in our results.

yes. So there's a big margin there, give or take fifteen percent. I guess you get just graph components, right? Do you have a sense of what about the graph algorithms makes one more successful than another?

So the T H, I think, IT all from everyone, most likely because of its in hand ability to capture information from largest, from neighbors, like immediate neighbors, perform like, follow our neighbors, like two of three half neighbors. I think that that would have have to be the 呃 reason that the topological away graph volume network can fleg out， performs all. yeah.

And do you have a thought about whether not this is not a world class performance? Or you know, if you spent A A lot more effort on algorithms, would you find some other solution? Or would you invest in feature engineering? I guess, how would you move IT forward if you were going to .

pursue this more? Uh yeah definitely this this was like a very time constraint experiments and and research, I would say, right. So this at least still a lot of Operations, uh, for improvement and like making the real Better.

I think a few paper that have cited this research kind of like have Better then we what we have right now, right? So they don't talk of what we do and then modify a few few things and then maybe like use a new algorithm that that have come up after like what we after we did our paper. Uh, so new algorithms for graph, a new relations are coming up.

So there are papers that side us have that had Better results than hours. So definitely there is room improvement. And there is also like more into, like this idea, a feature engineer, right? We have like how many features, six features that are very simple features, and we don't even into taking into account like the images of the flu plan.

So graphs are very useful in robotics, and your paper is one of many that establishes that. But robotics is also a pretty big field of a lot of specializations. Do you have any advice for an inspiring robotics? How important is IT for, uh, graphs and graph theory to show up on their syllabus?

Even in my life, uh, we do a lot of new networks for robots. For example, I can, I can talk about one, the one colleagues rearch. What he does is represent the whole house as a graph, and not just like a regular graph that we did like, but but like a higher ticket graph, right?

The higher the graves, look, something like that. The root node. And from the root note you have like the first hiera of rules, the root would represent the maybe an apartment.

And then inside the apartment there are rooms, right? So from the root node apartment, we have like a the first higher acy of rooms. And then inside the rooms you might have, like, table, and then pay, right.

And then that would form like another level of herself. On the table, you might have objects, right? A pen, a mouse, a keyboard are left up. And so he left just the higher graph to plan specifically for tax, tax and motion planning in robotics and rock s to the study of life. If you are given a task, they are safe.

And then his he what he dog is like, leverage this, uh, higher ticket grass to run a plan that most objects from one room to in the most efficient way. That's one example of like leverage graphic structures in the bodies. The other one we talked about earlier rates.

So this idea of pasta ding. Is all about rash even in even in like occupancy grade to plan a path from whether robot is to, let say, like a goal location, right? A lot of the pass spending algitha constitutes of graph algorithms.

For example, we ever as tighter as algorithms m, which is a clash algorithm, right, to plan a shortage pass from where the robot is toward to wants to go through, right? Maybe want to go from a, from a tour to a microwave inside the room, right? So you're represent the whole, whole, whole thing as an occupant's agreed, where each great sale would be a node and then you can traverse from one great day to another.

You have an age and you can level dale or a star algorithm gradually, especially ally, to plan the most efficient path from, whether about is to wait, wants to go to. So there's a lot of past planning that leverages graph algorithms. Ms, just talk about few. But yeah and graph and grass here and grass altham, they are like really what in newboy? S so definitely, if you are looking into a starting or like learning robot es, starting with like having like this idea and the concept of graph and graph theory would definitely be be a plus, and then is a must, I would say.

to what to great do you think of robots as agents may be even having their own utility functions.

a lot of the problems I was say, I mean, even like all a lot of the problems, even like maybe almost all of the problems can be founded in some kind of optimization problem, right? Uh, where like you have some optimization uh function that you are trying to either mini ze or maximize that already m is an optimizing problem, right? So you're trying to obtain a plan that minimizes the distance that robot travels from status le.

And that is all this bad. A lot of the problems can definitely be formulated in terms of like having a utility function, and then the result that you are trying to obtain. Me, if I were trying to minimize that utility function, or or maxims that util function, right?

For example, in robotics, a lot inforcements learning airline east twenty days, you have a reward and that the robot might get if IT, which is cold, and then there might be a negative reward. If IT, like stars wandering around more than IT needs to something like that, right? Once he finds the world, there might be a large, we were. So a lot of reinforcement alarm, reinforcement learning algorithms are trying to optimize how to get the best, or like how to get maximum rewards, and then trying to direct the robots behavior to us, getting this maximum expected to ours.

I know this is papers from a few years ago, is just one faster of your research. What are you most excited about today?

I'm excited about robotics, right? I'm doing the research and robotics. So a lot of the things that I think about right now is any and uncertainty, right? So what might not have all the information that he needs about the environment, right?

A lot of the part of the environment might be unknown and uh, sensor readings might miss leaders because they are noisy, right? Uh, you have to take into account a lot of those things while you are trying to make the robot behave the way you want and say humans would do. To that end, I mean, i'm excited about how to interview this idea of in respect and in robot, the robot does something right and IT makes me IT might make mistakes or IT might perform badly.

And i'm interested in this idea that the robot has to realize when it's making mistakes or when IT is behaving early. I asked robot that in the living room to go face me a knife for a four and then IT goes into bathroom, is to realize that OK I went to the bathroom, I did not find the knife. Maybe that was, that was not the best thing to do. Write this idea of, like when he does something wrong, when he does something poorly, is to realized that he made a mistake or profound poorly, and then use that information to correct its behaviour. This, I like this idea that I have been interested for a long time, and a lot of my research like is care to us like how to get the techniques that are therefore ally sound and practically feasible that levers this idea of interest action to improve, improve beer, beer when IT is deployed in unknown environments, or robots, or or environment IT has never seen before, or if if IT has seen before, IT might not have food access. To write on those kind of things here.

I presume introspection require some form of feedback. How else would you know that? You know, you need to find a mistake and where the mistake is, right? What form does that feedback take?

If I asked the robot to, again, like the same example, I bring a fork in the robot ge in living room, they say, IT goes to first, goes to bathroom, and then IT explores the bathroom and then realize that there is no for work. Right now he has to go to look, go look somewhere else, right? Maybe IT goes to bedroom and then realize that, okay, bedroom, there's no work. And until, like, IT finds, like finally goes to the kitchen and then finds the for now, after he finds the work, IT can now like left with the information is gathered from while travelling from the the living room to the bathroom to the bedroom and find out the kitchen to maybe like. So now once he has this more information at about the environment now, IT can make me think about what IT would have done if he had all the information beforehand, and then generate a new plan there. With this new plan, IT would have a Better plan, right? Because once IT has reached teacher and then found the fog, the Better plan would be to test go from living room to the kitchen directly right, and then compared this new plan with the old plan that is actually executed while finding the four right and that kind of like is an intuition for building this, uh, interest person that I leverage .

at my research. I've got the intuition of IT in hand. You know, i'm the robot.

Once i've got the fork, I can rethink. Oh, I should have just come directly here. yes. Are there popular methodologies to do that? How do you express that mathematically？

Uh, so mathematic, I an I mean, one thing that we do is in terms of the performance, right? So here the performance would maybe like be represented in terms of the distance that are about travel to find the object. Once you compare the actual distance that robot travelled to find the work, and then compare that a new plan, new imaginary plan, come to the distance from living room to the kids, and to find the for, like some kind of approximate distance the robot would have needed to travel.

This is quantitative measure of how much Better IT would have done and then we can build on top of that, can compare different plans IT would have generated from different loan model, right? So maybe like you use one model, some kind of model like any kind of model um that the toward can use based on his prior learning experience. And then robot uses that model, goes goes from living room to the batch room and then bedroom and then kitten to find the work.

If IT had used a different learning experience that I would have got on a different result. So OK you can kindly like reason about is different loan behavior or in robotics, where called policy, right? So policies which take the instead of the environment and gender, and right, that's a policy.

So if if the robot has multiple policies, IT can interrupt about what other policies could have done. That's kind of like one of the one of the ideas that are leveraging my research. So when to leverage the is existing knowledge and when to like try out new things, a lot of that that can be built up on this idea of, uh, intractable, where all the robot is kind of constantly monitoring is the behavior. And then I didn't find where would they perform Better or not?

I will. First rap of question then you being in the field of robotics, do you feel comfortable predicting anything in the next couple years? Where is IT going and how might IT impact everyday alive?

I don't have I I don't have a good answer, I guess. right? Robotic as a field is quite whole, and people considered like robotics to be moving very slowly, right? Because it's hard, you know, right? Robotics is hard, and this is very difficult.

And hard to get a robot to do things intelligently, right, is very, very difficult. So robot as a field is sometimes, I think, under appreciated because is not moving very as fast as people think we are compared to like muslim learning, like computer vision and alphy, right? You see the boom, right?

I think people and researchers are still like expecting a lot of the work to return before, like there is like a very useful and trust with robots out in the while, right? And I think I think we still have a lot of work to do in the bodies to make that happen. A lot of the researchers in this area are also like have a visions of when are we getting this trust with the Roberts out in the violent.

I feel like there's a long way to go and everyone contributing every everybody is is contributing in this like very small part, uh, an island of their own. I think I might take a wise to like connect those islands and then get like a really intelligent robot that you can trust and not worry about a lot. I think I am excited about the future in in that things because there's a lot of work to be done before we can read there.

And and that's what exactly you know like a lot of the work need to be needs to be done and and doing the work. A lot of people, people are doing their work. The future is bright, although IT may seem a little far.

Yeah and is there anywhere listeners can follow you online?

Uh, yes, so I am available on linking. I don't use a social media alive on lindon if you want to connect on on lindon sounds good.

Well, links in the show knows for this news, want to follow up. Thank you so for taking the time to come and and show your work.

And thank you guys. This was a very really impressive conversions, and thank you for the opportunity.