For people living with Asperger's syndrome, every social interaction can be a challenge. Keeping up with conversations can be especially difficult, since they have trouble interpreting the meaning of nonverbal communication (such as gestures and facial expressions). Two MIT researchers have set out to make these interactions less difficult.
Using wearable technology and deep learning AI systems, they have developed a tool that could one day act as a real-time virtual social coach.
In a recently published paper, MIT graduate student Tuka Alhanai and doctoral candidate Mohammad Ghassemi describe an AI system that uses specialized algorithms to analyze audio, text transcripts, and physiological signals to help determine the overall tone of a conversation in real time. The system runs on the Samsung Simband, a modular wrist-worn wearable with a wide variety of sensors and the ability to run custom algorithms on its hardware.
After training two algorithms on data collected by Simbands across 31 conversations, the research team found that the system could determine the overall tone of a conversation with 83 percent accuracy and provide more detailed “emotion scores” for targeted, specific five-second blocks of speech. The models predicted the mood of those blocks 7.5 percent better than other methods. Unlike most research in this field, this system was tested on real conversations, rather than simply asking participants to watch “happy” or “sad” videos.
The system is still in its early stages – it’s not the wearable social coach its creators envision. For now, the system only provides binary feedback for conversations as a whole, flagging individual interactions as either positive or negative. The Simband platform is another limiting factor, as it’s not yet commercially available.
But there is a clear path to further development. The researchers hope to find a way to use the system in commercial wearables like the Apple Watch, which would greatly expand the data available to the algorithms. With more data, the algorithms will learn and improve, which in turn will make the system more effective.
