The system is called Seeing AI and was built by Saqib Shaikh, a blind Microsoft engineer who wanted to become more involved in the world. Shaikh attended Microsoft’s //BUILD/ 2016 keynote talk yesterday to introduce the app, the first to be built on top of a new Microsoft framework known as Cognitive Services.
Microsoft Cognitive Services gives developers the power to access code that allows computers to see, hear, speak, understand and interpret needs delivered in natural communication. Seeing AI uses Cognitive Services to work out what is happening in the world and relay details to its user.
Shaikh controls Seeing AI with his phone and the Pivothead smart glasses. Seeing AI is triggered when Shaikh taps the side of his glasses, launching the app which attempts to work out what is present in his surroundings. Using Microsoft’s intelligence APIs to identify objects in the world, Seeing AI then uses text-to-speech to tell Shaikh what it has “seen.”
The app could give blind people like Shaikh a whole new way to experience the world. As a blind person, participating in conversations can be difficult. Body language is a vital but often overlooked aspect of communication but Shaikh has no way of experiencing it, making him hard for him to gauge reactions to what he says.
“When you’re talking to a bigger group, sometimes you can talk and talk and sometimes you think, is everyone listening really well, or are they half asleep? And you never know,” he explains in a video about Seeing AI. “The app can describe the general age and gender of the people around me and what their emotions are… which is incredible.”
In a demonstration of the technology, Shaikh wants to know the age and emotions of the people he is talking to. By tapping the side of his glasses, he launches Seeing AI which reveals he is speaking with a “40-year-old man with a beard, looking surprised” and a “20-year-old woman, looking happy.”
In another demonstration, Shaikh is seen sitting on a park bench. He doesn’t know what is happening but with a tap of his glasses he can find out. Seeing AI informs him “I think it’s a young girl with an orange Frisbee.”
The app can also deal with text. Shaik can have difficulty ordering things in restaurants when presented with a menu. Seeing AI can “read” the menu and then tell Shaik what’s on it. It can even understand commands such as “read me the headings” to give Shaik a better idea of what categories of food are available.
Seeing AI represents an angle of artificial intelligence that so far hasn’t seen much development. Apps built on frameworks like Microsoft Cognitive Services could prove to be instrumental in improving the lives of disabled people. By bringing different areas of technology together, new systems can be developed that redefine the limits of AI, helping people to experience their surroundings.
In Shaikh’s case, working at Microsoft has allowed him to interact with the world in ways that were impossible just a few years ago. As the technology develops, he hopes to be able to make it available to others, transforming lives and giving people access to more of their senses.
