How can a robot find its way in a city?

For humans it is easy to walk down the street, wait for the green light at the crossing, enter the grocery store to buy some apples and after that find our way back home.  But it is often these things, which we as humans consider simple, that robots have big troubles dealing with. The reason is that we are excellent in many tasks,  like recognizing where the sidewalk is,  avoid getting in the way of other pedestrians or finding an entrance to a shop, that we think they are easy. Robots, on the other hand, are not so good at understanding their environment. For instance, if we ask a robot simple questions like where we should cross the street, or about the color of the building in front, it will have a hard time giving an answer. 

For this reason, one of the research lines within the IURO project is to investigate how a robot can acquire this semantic level of understanding of the environment. We can do this by letting the robot learn about the different objects of the environment. For example, if we tell the robot that a specific region in an image is a tree, after multiple trials will be able to associate specific texture and color attributes to the concept of tree. In this way, the next time it sees these similar color and texture patterns,  it will be able to recognize it as a tree.  In practice, recognizing objects as cars, buildings, or surfaces like sidewalk, sky is a challenging task, because how they look varies with the viewpoint, the light conditions and many other factors, making it hard for robots to generalize.

Below we see a video of the robot building a semantic map of the street as it moves down the sidewalk, where different colors represent different categories. Of course,  it doesn't always get it right, and sometimes the robot hallucinates and sees cars inside of  buildings. 

Of course, perceiving the environment is just one step towards fully understanding the environments. In the future we will talk about other concepts such as planning, tracking and interaction, which will help us understand how a robot can find its way in a city.