Haptic Captioning - Using Natural Language Models to Design Haptic Experiences

From simple vibrations to touchable holograms in midair, haptic technology enables us to feel virtual or remote content and people. The impact of haptics is currently hampered by the difficulty of designing it. Existing design tools allow brute-force exploration of the large haptic signal space (e.g., intensity, frequency, rhythm), but they lack computational models that guide this exploration. Recent work suggest that people have a rich language for haptic signals (e.g., “This vibration feels exciting like drums or heartbeat.”). Can we use natural language processing (NLP) models to guide haptic signal exploration? This proposal aims to answer this question by developing algorithms that can predict user language for a haptic signal i.e., haptic captioning. The results will shift the current design paradigm in haptics. The new paradigm will enable building an entirely new class of languagebased haptic design tools, and it will inform development of communication schemes, multisensory media, and device interactions that utilize our touch sensory abilities. In contrast to image captioning, collecting user descriptions of haptic signals is expensive as it requires users to feel the stimuli via specialized hardware in the lab. The idea is to develop the haptic captioning algorithms by leveraging natural language processing (NLP) models that are trained on large non-sensory text corpuses and adapting them to haptic sensory data using a smaller purpose-built haptic dataset.