Working around unexpected events
“Today, however successful they are, robots are still mainly standalone applications e.g. used in automated fabrication,” says Pieter Simoens, Professor at IDLab, an imec research group at Ghent University. “In that situation, they are often fixed on one spot or even caged to protect the people working in their environment. These robots take their input from their own sensors and from sensors in the environment and are able to react to a limited range of predictable events.”
But in the future, robots are bound to get more freedom to move and interact. There will e.g. be butler robots that move though buildings to assist us, autonomous vehicles and drones that find their way to deliver services and packages, or automated search and rescue actions, to name but a few.
“The more autonomy these robots get, the more they will run into unexpected events,” says Simoens. “They will move through environments that are forever changing: doors get locked that were open before, furniture has changed places, roads are partially blocked... And during their actions, they may come upon other robots which will inevitably result in conflicts.”
Free-roaming robots can only function properly if they are in continuous two-way communication with the smart environment, i.e. if they become part of the IoT. Receiving sensor information, but also signaling their own position and intention as a kind of moving, dynamic sensor. A butler robot in a smart home, e.g. will signal where it is heading so that no doors are inadvertedly closed. And the cleaning robot in that same smart home will signal where it has just scrubbed the floor, so that no butler robot slips on the wet surface.
Second, instead of stubbornly proceeding to their goal, the robots will continuously adapt their next steps, maybe taking a pause or a detour before proceeding. “We’re working on techniques where only high-level planning is done upfront,” says Pieter Simoens. “These more abstract tasks are then unpacked into more concrete tasks in the moment they have to be executed. So the concrete actions that move the robot are only planned when that action has to be taken and the immediate context is available.”
Third, robots and actuators collaborating in the same smart space will have to coordinate and negotiate on the best course of action. This can e.g. be solved by having the various robots and actuators send their plans to a common platform that will then resolve the conflicts, work out a coordinated plan, and report back to the robots.
Zora, ready to provide care services for elderly residents
The human factor
“The most unpredictable factor that robots will encounter in smart spaces are people,” adds An Jacobs, sociologist at SMIT, an imec research group at VUB. “If we want to enhance our smart environment with robots that are useful and even likable, we will need to understand – much better than today – how people will want their robots, which services they want from robots, and which interaction with robots they deem appropriate.”
“To design appropriate robotic services, we can start with the existing and extensive body of work on human computer interaction. But robots bring a new dynamic in play: physical interaction and context-based intelligence. So the R&D has to be extended into a whole new territory.”
One of the projects where the imec researchers validated their research involved designing a new type of robot targeted at the production lines of the Audi factory in Brussels. Together with a number of specialized companies, they designed a ‘cobot’, a collaborative robot that can assist its human colleagues in a much more flexible way than traditional manufacturing robots do. Says Jacobs: “One challenge we looked into was how to make flexible cobots sufficiently robust for use in noisy, dynamic settings such as a car manufacturing line. Secondly, we also investigated the human side of the equation: how do machine operators want to interact with robots? And how do they expect robots to help them with their daily operations?”
One of the key issues in human-robot interaction is teaching the robot what it can and cannot do. That typically requires advanced programming skills, which is a serious bottleneck for flexible use. But in their cobot project, the imec researchers and their industrial partners came up with a new way to quickly instruct robots, an intuitive and efficient method that can be used by every machine operator, and that allows them to flexibly cope with on-the-fly and complex production changes.
“To achieve this,” continues Jacobs, “we used multimodal sensor data from heat, depth and color cameras and electrical current sensors – as well as from a newly developed cheap torque sensor. Through these, the robot gets a detailed view on the factory environment and the operators. And thanks to a facial and gesture recognition engine, it is capable of recognizing its human co-workers and interpreting the operator’s gestures and movements; input on which it can then act.”
The cobot Walt at work in the Brussels Audi plant
Setting up living labs
To design their cobot, there was no way the researchers could predict all possible scenarios beforehand. So they took to using another innovative approach.