With the new software, the new program can learn a typical sequence of actions, such as how a meal is cooked, from viewing video sequences. The artificial intelligence, using the knowledge it has acquired, can then predict, in new situations, what the cook will do at which point in time going forwards.
Application of such technology, as it is perfected, will have many uses beyond cooking. According to the research head, Professor Jürgen Gall, the aim is for many computers to have the ability to look beyond the present: “We want to predict the timing and duration of activities — minutes or even hours before they happen.”
Smart home of the future
Examples given fit with the smart home model. Imagine a kitchen robot that can pass the ingredients just as they are needed and pre-heat the oven just at the right time, as well as warning the cook that a key preparation step has been missed with the meal. At the same time, the robot vacuum cleaner knows that this is not the right time to enter the kitchen and can alter its cleaning path and begins vacuuming a different room.
Computers and robots remain very poor at sensing even a second beyond the current time. The German researchers have developed self-learning software that has the ability to estimate both the time of occurrence and likely duration of a future activity periods of several minutes into the future. Later work will seek to extend this to a longer time period.
The algorithm is based on two aspects of machine learning. First is a recurrent neural network that predicts for a given sequence of inferred activities the remaining duration of the ongoing activity as well as the duration and class of the next activity. The second is a convolutional neural network, which predicts a matrix that encodes the length and the action labels of the anticipated activities.
Trial in the kitchen
With the cooking trial, the researchers used 40 different cooking videos. In each video, a cook prepared different salads. The typical length of each recording was six minutes and the standard range of actions was twenty. The videos were prescriptive, and clear as to the time the action started and how long each action lasted.
A computer running the software was shown the four hours of video. As this happened the algorithm began to learn which actions tended follow each other during the salad preparation task and for how long they tended to last for. This was more complicated that it might seem, since in each video the person preparing the salad had his or her own variations. Furthermore, each salad recipe differed.
In order to assess what had been learnt, the computer was then shown new videos of salad preparation. Here the computer was only shown the first 20 percent of each new videos, and was then requested to predict the remaining actions over the remaining duration of the food preparation time.
The video below, “Anticipating Temporal Occurrences of Activities”, shows the experiment in action:
How accurate is it?
The accuracy was found to be around 40 percent; however, this fell the longer into the future the software attempted to predict. However, even at three minutes the algorithm was still 15 percent accurate. The scientists see this as the first step and they talk of a new research area emerging: the field of activity prediction
The artificial intelligence development is to be presented at the Conference on Computer Vision and Pattern Recognition, which takes place between June 19-21 in Salt Lake City, U.S. The conference, which has been running since 1983, covers any topic that is extracting structures or answers from images or video or applying mathematical methods to data to extract or recognize patterns.
A white paper has also been issued, titled “When will you do what? – Anticipating Temporal Occurrences of Activities”, and this can be downloaded and read in full.
Essential Science
This article is part of Digital Journal’s regular Essential Science columns. Each week Tim Sandle explores a topical and important scientific issue. Last week we considered new research which has found that for those with coronary heart disease, physical activity is more important for weight loss in terms of life expectancy following diagnosis of the disease.
The week before we discovered that methane ice dunes have been detected on the surface of Pluto, by NASA spacecraft. The structures offer a new insight into the dwarf planet and with the differences in planetary structures within our solar system.
