Affective State Prediction from Smartphone Touch and Sensor Data in the Wild

Apr 30, 2022·
Dr. Rafael Wampfler
Rafael Wampfler
,
S. Klingler
,
B. Solenthaler
,
V. R. Schinazi
,
M. Gross
,
C. Holz
Abstract
We investigate the prediction of affective states — including valence, arousal, and stress — from multimodal smartphone data collected in naturalistic conditions. Using a longitudinal study in which participants completed ecological momentary assessments while carrying instrumented smartphones, we collected rich streams of touch dynamics, motion sensor data, usage patterns, and environmental context. We develop and evaluate deep learning models that fuse these heterogeneous signals, demonstrating that multimodal fusion substantially outperforms single-modality baselines. We analyze the relative contribution of individual modalities, the effect of contextual factors on prediction accuracy, and challenges of individual variability in in-the-wild affective state prediction.
Type
Publication
In Proceedings of the Conference on Human Factors in Computing Systems (CHI), New Orleans, USA, Article 403
publications
Affective Computing Mobile Sensing Multimodal Learning Emotion Recognition In-the-Wild
Dr. Rafael Wampfler
Authors
Dr. Rafael Wampfler
Senior Researcher & Lecturer

I am a Senior Researcher & Lecturer at the Computer Graphics Laboratory of ETH Zurich, and a Research Consultant at Disney Research. I lead the Digital Character AI team at CGL and, as PI of an SNF Ambizione grant, a line of generative models that produce expressive 3D facial animation from speech and text under explicit control of emotion and expression. My research interests include conversational digital characters, generative animation, affective computing, human-computer interaction, and applied machine learning.

My vision is to create intelligent digital humans that can naturally communicate, understand, and support people across domains such as education and mental health. My research focuses on multimodal artificial intelligence for interactive digital humans, developing models that combine large language models, affective computing, and data-driven animation to create embodied conversational agents endowed with autonomous agency, consistent values, and beliefs.

My work bridges machine learning, human–computer interaction, and computer graphics to enable AI systems such as Digital Einstein and interactive patient avatars for psychotherapy training and health education.