A multimodal corpus of Human-Human and Human-Robot conversations including synchronized behavioral and neurophysiological recordings

Overview & Synopsis

This paper presents a unique corpus intended to be shared with the community at the crossroad of linguistics and human-robot interactions (HRI). It is the result of an interdisciplinary collaborations within Cognitive Sciences combining social neu-rosciences, computational sciences, robot-ics and linguistics. It was recorded when brain activity of 25 participants was scanned using functional Magnetic Resonance Imaging (fMRI) while having uncon-strained conversations with a fellow human or a conversational robotic head. Behaviors from the participant and the conversant were recorded synchronously (speech, eye-tracking, head and face movements). Manual and automatic analysis of these behaviors provide rich sets of data for the analysis of both behaviors and neurophysiological responses. Examples of results obtained with this corpus are provided.