Language is more than words. Just think about the meaning that can reside in silence. Embodied Conversational Agents and Social Robots that are capable of engaging in a natural language dialogue with people cannot just stick to words and phrases when communicating with their human interlocutors. As they are embodied - in virtual space, in plastic or metal - often resembling human forms, they also need to use their heads, faces, hands, arms and body to communicate when they do not want to look too awkward. Looked at from the other side, the conversational skills required of these agents and robots comprise the perception and understanding of the nonverbal cues that accompany the speech of their conversational partner. They need to understand the silence as a place to put in a backchannel, a place to start a sentence, as a hesitation, or as expressing that the person they are talking to is overwhelmed and at a loss of words.

In this talk we will present a short overview of almost two decades of work on Attentive Virtual Humans (and Robots). In this work, we have frequently relied on corpora of human-human interaction as a basis to build conversational agents. In this talk we will focus on some issues in the collection and annotation of data as well. In particular we will talk more about the nature of non-verbal signalling and how this raises questions about the methodology that has been de rigueur in most work of the past decades.