egoEMOTION: Egocentric Vision and Physiological Signals for Emotion and Personality Recognition in Real-World Tasks

Dec 1, 2025·
M. Jammot
,
B. Braun
,
P. Streli
Dr. Rafael Wampfler
Dr. Rafael Wampfler
,
C. Holz
Abstract
We present egoEMOTION, a new dataset and multimodal deep learning architecture for emotion and personality recognition in real-world tasks, combining egocentric vision from a head-mounted camera with physiological signals including heart rate and electrodermal activity. Unlike existing benchmarks collected in controlled laboratory settings, egoEMOTION captures naturalistic affective states during everyday activities, enabling in-the-wild evaluation. We introduce a unified fusion architecture that jointly encodes visual context from the wearer’s first-person perspective and concurrent physiological responses, advancing both fusion strategies and reproducible research through an open dataset contribution.
Type
Publication
In Conference on Neural Information Processing Systems 2025 (Datasets and Benchmarks, NeurIPS)
publications
Affective Computing Egocentric Vision Emotion Recognition Personality Recognition Physiological Signals Multimodal Learning
Dr. Rafael Wampfler
Authors
Dr. Rafael Wampfler
Senior Researcher & Lecturer

I am a Senior Researcher & Lecturer at the Computer Graphics Laboratory of ETH Zurich, and a Research Consultant at Disney Research. I am leading the Digital Character AI projects at CGL. My research interests include conversational digital characters, affective computing, human-computer interaction, and applied machine learning.

My vision is to create intelligent digital humans that can naturally communicate, understand, and support people across domains such as education and mental health. My research focuses on multimodal artificial intelligence for interactive digital humans, developing models that combine large language models, affective computing, and data-driven animation to create embodied conversational agents endowed with autonomous agency, consistent values, and beliefs.

My work bridges machine learning, human–computer interaction, and computer graphics to enable AI systems such as Digital Einstein and interactive patient avatars for psychotherapy training and health education.