Dr. Rafael Wampfler

Digital Einstein

Tue, 01 Jun 2021 00:00:00 +0000

Overview

Digital Einstein is a flagship embodied conversational agent that brings the historical figure of Albert Einstein to life through real-time multimodal AI interaction. The system combines speech recognition and synthesis, facial animation, gesture control, and a cognitively grounded language understanding pipeline to deliver immersive, personality-consistent conversations.

Digital Einstein serves three interconnected roles:

Research platform — a testbed for studying human–agent interaction in constrained embodied settings, yielding insights on affective computing, personality modeling, dialog act classification, and conversational AI.
Education platform — a live demonstration of conversational AI and multimodal deep learning deployed in university events, science outreach, and public engagement.
Public engagement tool — reaching thousands of visitors globally at scientific conferences, tech summits, museums, and public events, generating sustained international recognition for ETH Zurich.

Motivation

How can AI systems convincingly portray a well-known historical personality — someone whose knowledge, values, and speaking style are culturally established — in real-time dialogue with arbitrary members of the public? This challenge crystallizes core problems in interactive AI: maintaining factual and characterological consistency, adapting dynamically to unpredictable user inputs, and delivering a compelling embodied experience at scale.

Digital Einstein was conceived as both a scientific challenge and a communication vehicle: making abstract advances in AI tangible for general audiences while simultaneously driving rigorous research on the underlying problems.

Approach

The system is built on a full-pipeline architecture described in the SIGGRAPH 2025 paper “A Platform for Interactive AI Character Experiences”. Key components include:

Perception layer: Real-time speech recognition via Microsoft Azure Speech Services and multimodal input processing through a webcam-based vision pipeline, including face detection, user characterization, head pose estimation, and re-identification.
Cognitive reasoning layer: Knowledge-grounded dialogue management with integrated response generation, powered by GPT-4.1 mini, featuring dynamic personality infusion that adapts outputs to user-selectable archetypes: Digital Einstein, Rude Bulldozer, Drama Volcano, Zen Master, and Hashtag Prophet.
Animation synthesis: Data-driven facial animation synchronized with speech output using NVIDIA Audio2Face, blended with emotion-conditioned expressions, and complemented by a curated library of motion-captured body animations categorized by avatar state.
Embodiment: A stylized Albert Einstein avatar rendered in Unity on a 65-inch display, integrated into a themed early-20th-century physical environment with spatial audio, a hidden microphone, and physical personality sliders built from potentiometers and an Arduino.

The SIGGRAPH Asia 2024 demonstration paper “Immersive Conversations with Digital Einstein: Linking a Physical System and AI” details the physical installation setup, including the integration of an animatronic head with the real-time AI pipeline at the Tokyo venue.

Key Results

Digital Einstein has been demonstrated at over 20 major events worldwide, including:

SIGGRAPH Asia 2024 (Tokyo, Japan) — Emerging Technologies
SIGGRAPH 2025 (Vancouver, Canada)
GITEX Global 2024 & 2025 (Dubai, UAE) — Swiss Pavilion
World Economic Forum 2024 & 2026 (Davos, Switzerland)
Berlin Science Week 2025 (Berlin, Germany)
Swiss Re Resilience Summit 2024 (Rüschlikon, Switzerland)
Microsoft Initiative to Advance AI Diffusion in Switzerland 2025 (Berne)
After the Algorithm Festival 2026 (Zurich, Switzerland)

The project has generated sustained international media coverage and public interest, positioning ETH Zurich as a world leader in embodied conversational AI.

Swiss Ambassador to the UAE, Arthur Mattli, interacting with Digital Einstein at GITEX Global in Dubai.

Learn More

Primary Publications

R. Wampfler, C. Yang, D. Elste, N. Kovačević, P. Witzig and M. Gross (2025). A Platform for Interactive AI Character Experiences. Proceedings of the SIGGRAPH Conference Papers ‘25 (Vancouver, Canada, August 10–14, 2025), pp. 1–11.

R. Wampfler, N. Kovačević, P. Witzig, C. Yang, M. Gross (2024). Immersive Conversations with Digital Einstein: Linking a Physical System and AI. In SIGGRAPH Asia 2024 Emerging Technologies (SA ‘24) (Tokyo, Japan, December 3–6, 2024).

Affective Computing & Emotion Recognition

Tue, 01 Jan 2019 00:00:00 +0000

Overview

This research thread develops deep learning architectures for predicting human emotional and cognitive states from rich, naturalistic data streams. Unlike laboratory-controlled setups, our systems operate “in-the-wild” — on real devices, in real environments, with real users — addressing the full complexity of affective computing at scale.

Motivation

Affective computing — the capacity of machines to detect, interpret, and respond to human emotions — is a foundational capability for human-centric AI. Yet most academic benchmarks rely on controlled, acted datasets that poorly predict real-world performance. Building systems that genuinely work in naturalistic settings requires confronting three fundamental challenges:

Domain adaptation: Affective signals vary enormously across individuals and contexts; models must transfer gracefully.
Uncertainty estimation: Emotion recognition inherently involves ambiguity and subjectivity; systems must quantify and communicate their confidence.
Scalability: Continuous affective sensing must operate on resource-constrained mobile and edge devices.

Approach

Multimodal Fusion

Our work leverages a broad set of input modalities, combining them through transformer-based and convolutional architectures:

Smartphone touch and sensor data: Stylus pressure, touch dynamics, accelerometer, and gyroscope signals during naturalistic task completion
Biometric data: Heart rate, skin conductance, and other physiological signals from wearables
Egocentric vision: First-person video from wearable cameras, capturing the user’s visual environment
Typing behavior: Smartphone keyboard dynamics as a passive indicator of affective and personality state

Semi-Supervised and Self-Supervised Learning

Given the difficulty and cost of obtaining large labeled affective datasets in natural settings, we exploit semi-supervised learning strategies that leverage abundant unlabeled data. This improves generalization without requiring exhaustive annotation.

egoEMOTION (NeurIPS 2025)

The most recent and ambitious contribution is egoEMOTION, presented at NeurIPS 2025 (Datasets and Benchmarks track). This work combines egocentric vision and physiological signals into a unified multimodal architecture, advancing both fusion strategies and providing a new reproducible benchmark dataset. egoEMOTION addresses the challenge of predicting emotion and personality from the wearer’s own perspective, a naturalistic setting of growing relevance as wearable cameras become ubiquitous.

Personality Recognition from Typing

Beyond momentary emotions, we have also developed systems for personality trait recognition from passive smartphone typing dynamics. This work (IEEE Transactions on Affective Computing, 2023) demonstrates that stable personality traits leave measurable signatures in everyday smartphone interactions, enabling passive, continuous personality inference.

Key Results

Demonstrated state-of-the-art in-the-wild affective state prediction from smartphone sensors across multiple CHI publications
Published a new egocentric multimodal emotion and personality benchmark (NeurIPS 2025)
Showed that semi-supervised learning substantially closes the gap between labeled and unlabeled-data performance
Developed personality trait recognition from typing dynamics achieving strong classification performance on real-world data

Publications

M. Jammot, B. Braun, P. Streli, R. Wampfler and C. Holz (2025). egoEMOTION: Egocentric Vision and Physiological Signals for Emotion and Personality Recognition in Real-World Tasks. In Conference on Neural Information Processing Systems 2025 (Datasets and Benchmarks, NeurIPS), pp. 1–12.

N. Kovačević, C. Holz, M. Gross and R. Wampfler (2024). On Multimodal Emotion Recognition for Human-Chatbot Interaction in the Wild. In Proceedings of the 26th International Conference on Multimodal Interaction (ICMI ‘24), San Jose, Costa Rica, November 4–8, 2024.

N. Kovačević, C. Holz, T. Günther, M. Gross and R. Wampfler (2023). Personality Trait Recognition Based on Smartphone Typing Characteristics in the Wild. IEEE Transactions on Affective Computing, pp. 1–11, 2023.

R. Wampfler, S. Klingler, B. Solenthaler, V. R. Schinazi, M. Gross and C. Holz (2022). Affective State Prediction from Smartphone Touch and Sensor Data in the Wild. Proceedings of the Conference on Human Factors in Computing Systems (CHI), New Orleans, USA, April 30–May 5, 2022, pp. 1–14.

R. Wampfler, S. Klingler, B. Solenthaler, V. R. Schinazi and M. Gross (2020). Affective State Prediction Based on Semi-Supervised Learning from Smartphone Touch Data. Proceedings of the Conference on Human Factors in Computing Systems (CHI), Virtual, April 25–30, 2020, pp. 1–13.

N. Kovačević, R. Wampfler, B. Solenthaler, M. Gross and T. Günther (2020). Glyph-Based Visualization of Affective States. Eurographics/IEEE VGTC Symposium on Visualization (EuroVis), Virtual, May 25–29, 2020, pp. 121–125.

R. Wampfler, S. Klingler, B. Solenthaler, V. R. Schinazi and M. Gross (2019). Affective State Prediction in a Mobile Setting using Wearable Biometric Sensors and Stylus. Proceedings of the International Conference on Educational Data Mining (EDM), Montréal, Canada, July 2–5, 2019, pp. 224–233.

Personality & Cognitive Architectures for Conversational Agents

Sat, 01 Jan 2022 00:00:00 +0000

Overview

Standard large language models (LLMs) are stateless by design: each response is generated without a persistent sense of self, stable values, or accumulated memory of prior interactions. This makes them ill-suited for applications requiring a believable, consistent agent personality — such as embodied historical figures, therapeutic companions, or narrative characters. This project develops deep learning-based cognitive frameworks that address this limitation.

Motivation

For conversational agents to be genuinely useful in education, mental health, entertainment, and public engagement, they must do more than generate fluent text. They must maintain:

Characterological consistency: A stable personality that users recognize and can model predictively
Value alignment: Responses grounded in a coherent set of beliefs and priorities
Epistemic coherence: Awareness of what the agent knows and does not know
Emotional expressivity: Affect that reflects the agent’s personality and conversational context

Achieving all of this while remaining interpretable and safe requires principled architectural choices rather than prompt engineering alone.

Approach

The BEE Cognitive Framework (IVA 2025)

The Belief-Value-Aligned, Explainable, and Extensible (BEE) cognitive framework provides a structured architecture for conversational agents. BEE decomposes agent cognition into explicit modules:

Belief management: A persistent, queryable representation of the agent’s knowledge and world model
Value alignment: A constraint layer that ensures generated responses conform to the agent’s ethical commitments and personality
Explainability: Transparent reasoning traces that allow developers and supervisors to audit agent behavior
Extensibility: A modular design that supports new knowledge domains and personality profiles without retraining

BEE received the Best Paper Honorable Mention at IVA 2025.

Joint Personality-Emotion Framework (IVA 2025)

The Joint Personality-Emotion Framework unifies personality modeling (using Big Five trait representations) with momentary emotional state tracking in a single learned architecture. By jointly modeling personality and emotion — which are deeply intertwined in human behavior — the system produces responses that feel emotionally authentic and characterologically stable. Contrastive learning strategies decouple emotion from semantic content, allowing the same underlying personality to express itself across diverse emotional registers.

Personality Infusion via Chatbot Interactions (CUI 2024, IMWUT 2024)

We have extensively studied how personality manifests in conversational AI systems. Key findings include:

What personality dimensions does GPT-3 express during human-chatbot interactions, and how do these map to the Big Five personality taxonomy (IMWUT 2024)
Dynamic personality infusion — how to modulate expressed personality in real time by conditioning language model generation on personality embeddings (CUI 2024)

Narrative Agents and Emergent Storytelling (AAAI AIIDE 2025)

In the domain of interactive narrative, we developed a Dynamic Cognitive Framework for Guided Emergent Storytelling that enables narrative agents to pursue coherent story arcs while remaining responsive to player input. The framework uses explicit belief-desire-intention representations to steer language model generation toward narratively coherent outcomes without sacrificing interactivity.

Key Results

BEE framework achieved Best Paper Honorable Mention at IVA 2025
Joint Personality-Emotion Framework demonstrated stable personality maintenance across extended dialogues with diverse user populations
IMWUT 2024 paper provided the first large-scale empirical characterization of GPT-3’s expressed personality dimensions
Dynamic narrative framework enabled coherent 30+ turn story experiences with novel player choices

Publications

N. Kovačević, C. Holz, M. Gross and R. Wampfler (2025). A Joint Personality-Emotion Framework for Personality-Consistent Conversational Agents. In Proceedings of the 25th International Conference on Intelligent Virtual Agents (IVA ‘25), Berlin, Germany, September 16–19, 2025, pp. 1–9. Best Paper Honorable Mention.

C. Yang, M. Gross and R. Wampfler (2025). BEE: Belief-Value-Aligned, Explainable, and Extensible Cognitive Framework for Conversational Agents. In Proceedings of the 25th International Conference on Intelligent Virtual Agents (IVA ‘25), Berlin, Germany, September 16–19, 2025, pp. 1–9. Best Paper Honorable Mention.

C. Yang, M. Gross and R. Wampfler (2025). Steering Narrative Agents through a Dynamic Cognitive Framework for Guided Emergent Storytelling. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE), Edmonton, Canada, November 10–14, 2025, pp. 1–11.

N. Kovačević, C. Holz, M. Gross and R. Wampfler (2024). The Personality Dimensions GPT-3 Expresses During Human-Chatbot Interactions. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, ACM, vol. 8, no. 2, 2024, pp. 1–36.

N. Kovačević, T. Boschung, C. Holz, M. Gross and R. Wampfler (2024). Chatbots With Attitude: Enhancing Chatbot Interactions Through Dynamic Personality Infusions. Proceedings of the 6th International Conference on Conversational User Interfaces (CUI), Luxembourg, July 08–10, 2024, pp. 1–16.

Facial Animation Synthesis

Mon, 01 Jan 2024 00:00:00 +0000

Overview

Realistic, expressive facial animation is a critical component of embodied conversational agents. For AI characters to communicate naturally, their facial movements must be synchronized with speech, emotionally consistent, and computationally efficient enough for real-time use. This project develops deep learning architectures that achieve all three goals.

Motivation

The challenge of generating high-quality 3D facial animation from text or speech involves several competing requirements:

Expressiveness: Facial motion should convey the speaker’s emotional state convincingly
Synchronization: Lip movements must match phoneme timing precisely to avoid the uncanny valley
Consistency: Emotional expressivity should be decoupled from semantic content, allowing the same phrase to be delivered in multiple emotional registers
Efficiency: Systems deployed in interactive agents must run in real time on consumer hardware

Prior approaches either sacrifice expressiveness for speed, or require enormous training data and computation. Our work addresses both efficiency and expressiveness simultaneously by rethinking how deep learning encodes linguistic and acoustic structure.

Approach

PhonemeNet (MIG 2025)

PhonemeNet applies a transformer pipeline specifically designed for the phoneme-level structure of speech. Rather than treating the speech signal as a raw audio waveform or frame-level features, PhonemeNet operates at the level of phonemes — the fundamental units of speech that determine lip shape. This problem-specific inductive bias yields both improved accuracy and computational efficiency compared to architectures that ignore linguistic structure.

PhonemeNet takes text input, extracts phoneme sequences, and generates corresponding 3D facial blendshape sequences that are synchronized with speech audio. The pipeline achieves real-time performance on standard hardware, making it suitable for deployment in interactive embodied agents.

PhonemeNet received the Best Paper Honorable Mention at the 18th ACM SIGGRAPH Conference on Motion, Interaction, and Games (MIG 2025).

EmoSpaceTime (MIG 2024)

EmoSpaceTime addresses the problem of emotionally expressive 3D speech animation through a contrastive learning strategy. The core insight is that facial animation should be factorized into two independent components: emotion (how the speaker feels) and content (what the speaker is saying). By learning to decouple these in a shared embedding space, EmoSpaceTime enables:

Transfer of emotional style between speakers
Consistent emotional expressivity across different sentences
Fine-grained control over emotional intensity at inference time

The resulting animations are both emotionally coherent — the emotion is consistent throughout an utterance — and semantically coherent — lip synchronization is accurate regardless of the emotional style applied.

Key Results

PhonemeNet achieves real-time text-driven facial animation with best-in-class lip synchronization accuracy — Best Paper Honorable Mention at MIG 2025
EmoSpaceTime demonstrates that contrastive decoupling of emotion and content significantly improves expressive quality while maintaining temporal coherence

Publications

P. Witzig, B. Solenthaler, M. Gross and R. Wampfler (2025). PhonemeNet: A Transformer Pipeline for Text-Driven Facial Animation. Proceedings of the 18th ACM SIGGRAPH Conference on Motion, Interaction, and Games (MIG ‘25), Zurich, Switzerland, December 3–5, 2025, pp. 1–11. Best Paper Honorable Mention.

P. Witzig, S. Solenthaler, M. Gross, R. Wampfler (2024). EmoSpaceTime: Decoupling Emotion and Content through Contrastive Learning for Expressive 3D Speech Animation. In Proceedings of the 17th ACM SIGGRAPH Conference on Motion, Interaction and Games (MIG ‘24), Arlington, USA, November 21–23, 2024.

Dialog Act Classification

Mon, 01 Jan 2024 00:00:00 +0000

Overview

For a conversational agent to respond appropriately, it must understand not just what a user says, but why they said it — the communicative intent behind their utterance. Dialog Act (DA) classification is the task of categorizing utterances by their function in conversation (e.g., question, assertion, greeting, request, clarification). This project develops multimodal dialog act classifiers tailored for interactions with digital characters.

Motivation

Standard dialog act classification systems are trained on text transcriptions alone. In real-world interactions with embodied agents, however, users communicate through a rich combination of speech prosody, gaze, gesture, and lexical content. A question delivered with rising intonation carries different meaning than the same words spoken flatly; a greeting accompanied by eye contact differs from one delivered distractedly.

For digital characters that must respond naturally in real time, dialog act classification must therefore be multimodal — integrating acoustic, linguistic, and where available, visual signals — and must operate with low latency to support interactive response times.

Approach

Our multimodal dialog act classifier integrates:

Lexical features: Encoded via transformer-based text encoders fine-tuned on dialog corpora
Acoustic features: Prosodic signals including pitch, energy, and speech rate, extracted from the raw audio signal
Temporal context: Conversation history modeling to resolve ambiguous acts through discourse-level context

The system is evaluated on naturalistic conversations with digital characters — a challenging setting because users frequently use fragmented, spontaneous speech rather than complete, grammatical sentences. The classifier is optimized for both accuracy and latency, enabling real-time use within the Digital Einstein pipeline.

Key Results

Demonstrated that multimodal integration (text + acoustic features) significantly outperforms text-only baselines for dialog act classification in digital character conversations
Achieved real-time classification latency compatible with interactive agent deployment
Provided insights into which dialog acts are most frequently misclassified in human-agent interaction, informing future system design

Publication

P. Witzig, R. Constantin, N. Kovačević and R. Wampfler (2024). Multimodal Dialog Act Classification for Conversations With Digital Characters. Proceedings of the 6th International Conference on Conversational User Interfaces (CUI), Luxembourg, Luxembourg, July 08–10, 2024, pp. 1–14.

Virtual Psychotherapist

Sun, 01 Jan 2023 00:00:00 +0000

Overview

Access to evidence-based psychotherapy remains severely limited worldwide — constrained by long waitlists, resource scarcity, and geographic disparities. The Virtual Psychotherapist project develops embodied conversational AI agents that complement clinical care by extending access, supporting therapeutic practice, and enabling scalable training.

This initiative is conducted in close collaboration with the University of Lucerne, providing clinical expertise and direct access to real therapy data.

Motivation

Two distinct but complementary challenges motivate this project:

Scaling patient access: Many individuals who would benefit from psychotherapy cannot access it due to cost, waitlists, or geography. AI-based companions that operate between sessions, provide continuous support, and conduct structured therapeutic conversations could meaningfully improve outcomes at scale.
Improving therapist training: Training clinicians in evidence-based interventions requires repeated practice with feedback — but opportunities for safe, standardized training are limited by ethical and resource constraints. Simulated patient systems powered by AI can provide unlimited deliberate practice in controlled settings.

System Architecture

Both applications are built on a shared platform combining:

Large language model-based dialogue: State-of-the-art LLMs for contextually appropriate, therapeutically grounded response generation
Retrieval-augmented generation (RAG): Responses grounded in evidence-based therapeutic literature, reducing hallucinations and ensuring adherence to clinical frameworks
Real-time psychological analysis: Parallel processing pipelines that extract facts, detect psychological flexibility processes, recognize emotions, and monitor safety in real time
Embodied avatar presentation: Synchronized speech synthesis and 3D avatar animation delivered through mobile and desktop applications
Clinician oversight: Structured analysis outputs accessible to supervising therapists, enabling human-in-the-loop clinical governance

Patient-Facing Application

The patient-facing component enables individuals to conduct therapeutic conversations with an embodied avatar between sessions. The system follows process-based therapy principles — particularly Acceptance and Commitment Therapy (ACT), an empirically supported approach targeting psychological flexibility through six core processes: acceptance, cognitive defusion, present-moment awareness, self-as-context, values clarification, and committed action.

Critically, the system is designed for clinical supervision, not autonomous intervention. All session data is structured and accessible to the supervising therapist, who can monitor progress and intervene as needed.

An evaluation against responses from professional psychotherapists demonstrated that the system’s responses were rated significantly higher on understanding, interpersonal effectiveness, collaboration, and ACT alignment — while emphasizing that clinical judgment and the therapeutic relationship remain irreplaceable.

Therapist Training Application

The training application enables psychotherapists to practice and refine therapeutic techniques through role-play interactions with a simulated patient. Key features include:

Clinically grounded patient simulation: Virtual patient behavior conditioned on profiles derived from real therapy sessions, covering a range of clinical presentations and scenarios (suicidality, resistance, heightened anxiety, therapeutic rupture, and more)
Real-time ACT fidelity feedback: An automated evaluator assesses each therapist utterance for adherence to ACT principles, providing immediate visual feedback and the option to retry alternative responses
Configurable scenarios: Therapists can select specific clinical scenarios to target their practice

A systematic evaluation across 49 therapy transcripts identified GPT-4o-mini as the optimal feedback model, achieving the closest alignment with human supervisor ACT fidelity ratings.

Safety and Ethics

Safety is a primary design constraint. The system includes:

Crisis detection: Explicit classification of suicidal ideation and self-harm signals, triggering immediate presentation of crisis resources and mandatory clinician review
Unsafe-interaction detection: Identification of conditions (e.g., active psychosis, mania) where LLM interaction may be counterproductive, with protocol-defined fallback responses
Non-autonomous design: The system is explicitly positioned as a complement to clinical care, not a replacement — structured to require and facilitate clinician oversight

Key Results

Patient-facing system responses rated significantly higher than human therapist responses by automated evaluation and expert psychotherapists across understanding, collaboration, and ACT alignment
Therapist training simulation rated as realistic by practicing psychologists; turn-by-turn feedback shown to increase therapist awareness of intervention choices
Automated ACT fidelity assessment achieves strong agreement with human expert ratings across 49 therapy transcripts

Research Partners

University of Lucerne

RehaBot

Mon, 01 Jan 2024 00:00:00 +0000

Overview

RehaBot is an embodied conversational agent designed to support patients in rehabilitation and home-care settings. The avatar represents a medical professional — capable of conducting structured patient interactions, administering assessments, and delivering health education — to help bridge the gap between in-clinic care and independent recovery at home.

This project is developed in collaboration with Inselspital Bern (University Hospital) and Bern University of Applied Sciences (BFH).

Motivation

Rehabilitation after medical treatment — whether from stroke, orthopedic surgery, cardiac events, or chronic disease — requires sustained patient engagement over weeks or months. Yet contact with healthcare professionals is necessarily episodic, leaving long gaps during which patients must self-manage. Lack of guidance, motivation, and timely feedback during these intervals is a major driver of poor rehabilitation outcomes and preventable hospital readmissions.

An embodied conversational agent that patients can interact with at home — to receive reminders, answer questions, conduct structured assessments, and provide health education — addresses this gap directly. By combining medical knowledge with empathetic communication and a human-like embodied presence, RehaBot aims to make professional-quality support continuously available between clinical appointments.

Approach

RehaBot integrates several complementary AI capabilities within a unified embodied avatar system:

Medical knowledge integration: Structured clinical knowledge relevant to the patient’s rehabilitation pathway, enabling accurate and safe responses to health questions
Conversational assessment: The ability to administer structured health questionnaires and functional assessments through natural spoken dialogue, adapting pacing and clarification to individual patient needs
Empathetic communication: Affective modeling that allows the agent to detect and respond to emotional signals in patient speech — frustration, discouragement, anxiety — with appropriate supportive responses
Health education: Accessible explanations of rehabilitation exercises, medication adherence, warning signs, and self-management strategies, adapted to the patient’s comprehension level
Patient-professional interface: Structured summaries of patient interactions accessible to supervising clinicians, supporting continuity of care and early detection of clinical deterioration

The system is built on the same core platform as the Digital Einstein project, enabling rapid deployment of new capabilities while maintaining consistent embodied presentation quality.

Key Results

Recent work exploring embodied conversational interfaces for personal health data reflection demonstrates that users who engage with health information through a conversational agent formulate significantly more specific and actionable health plans compared to traditional dashboard-based exploration. Embodied conversation lowers the cognitive burden of interpreting health data and supports a shift from passive data inspection to active health sensemaking.

Research Partners

Inselspital Bern (University Hospital of Bern)
Bern University of Applied Sciences (BFH)

MIND — Cognitive Health Monitoring

Wed, 01 Jan 2025 00:00:00 +0000

Overview

The MIND (Monitoring and Supporting Cognitive Health) project creates a mobile platform for the early detection of cognitive impairment in aging populations. Early identification of decline — before it significantly impacts daily life — enables timely intervention, supports independent living, and improves long-term outcomes for older adults and their families.

MIND is part of the Future Health Technologies 2 (FHT2) initiative, a major research program funded by the National Research Foundation Singapore (NRF) CREATE grant.

Motivation

Cognitive decline — from mild cognitive impairment to dementia — affects hundreds of millions of people worldwide. Current clinical detection relies predominantly on periodic in-clinic assessments that are episodic, costly, and often catch decline only after substantial impairment has occurred. A continuously operating, passive monitoring system that can detect subtle early warning signs in everyday behavior could transform the standard of care.

Two key insights motivate the MIND approach:

Movement and navigation encode cognition: Changes in how people navigate their environment — route choices, wayfinding strategies, spatial memory — are among the earliest and most sensitive indicators of cognitive decline. Analyzing GPS and sensor traces at scale can reveal these changes.
Language reflects cognitive health: Subtle changes in vocabulary, sentence complexity, topic management, and response latency in everyday conversation are measurable markers of cognitive change, detectable through automated language analysis.

Approach

MIND integrates three complementary AI systems into a unified mobile platform:

Large Geospatial Models (LGMs)

Large Geospatial Models analyze longitudinal GPS and mobility traces to detect patterns indicative of cognitive change. These include:

Route repetitiveness: Increasing restriction of daily movement range
Navigation errors: Unusual detours or disorientation in familiar environments
Mobility diversity: Changes in the variety of visited locations over time

LGMs are trained on large-scale mobility datasets and fine-tuned to identify individual-level deviations from baseline behavior.

LLMs for Conversational Cognitive Assessment

Large language models analyze conversation transcripts from daily interactions with the MIND app, detecting cognitive markers such as:

Reduced lexical diversity and increased use of filler words
Difficulty with topic maintenance and coherence
Slowed response generation and increased hesitation
Confusion or confabulation in response to everyday questions

These markers are tracked longitudinally to identify meaningful change relative to the individual’s own baseline.

Embodied Avatars for Guided Assessment and Training

An embodied conversational avatar conducts structured cognitive assessments — such as memory recall tasks, verbal fluency tests, and orientation questions — through natural spoken dialogue. The avatar also guides cognitive training exercises designed to maintain cognitive reserve.

The embodied format is critical: older adults are more likely to engage regularly with an interactive, socially present agent than with a text-based questionnaire or passive sensor. The avatar adapts its communication style to individual users, adjusting vocabulary, pacing, and support level to ensure accessibility.

Integration and Privacy

All three systems operate on a shared mobile platform with strong privacy protections. Data is analyzed on-device where possible, and users maintain full control over what is shared with care providers. The platform produces structured reports for geriatricians and primary care physicians, enabling early clinical intervention.

Significance

MIND represents a convergence of several research threads — geospatial AI, conversational AI, affective computing, and embodied interaction — applied to one of the most pressing health challenges of our time. By enabling early, passive, continuous monitoring, the platform aims to support successful aging in place and to delay or prevent the transition to care dependency.

Research Partners and Funding

National Research Foundation Singapore (NRF) CREATE — Future Health Technologies 2 (FHT2)
Bond University (Australia)
Collaboration with life sciences and medicine partners in Singapore

Talking to Your Data: Exploring Embodied Conversation as an Interface for Personal Health Reflection

Mon, 23 Mar 2026 00:00:00 +0000

PhonemeNet: A Transformer Pipeline for Text-Driven Facial Animation

Wed, 03 Dec 2025 00:00:00 +0000

Best Paper Honorable Mention — 18th ACM SIGGRAPH Conference on Motion, Interaction, and Games (MIG 2025)

egoEMOTION: Egocentric Vision and Physiological Signals for Emotion and Personality Recognition in Real-World Tasks

Mon, 01 Dec 2025 00:00:00 +0000

Steering Narrative Agents through a Dynamic Cognitive Framework for Guided Emergent Storytelling

Mon, 10 Nov 2025 00:00:00 +0000

A Joint Personality-Emotion Framework for Personality-Consistent Conversational Agents

Tue, 16 Sep 2025 00:00:00 +0000

BEE: Belief-Value-Aligned, Explainable, and Extensible Cognitive Framework for Conversational Agents

Tue, 16 Sep 2025 00:00:00 +0000

Best Paper Honorable Mention — 25th International Conference on Intelligent Virtual Agents (IVA 2025)

A Platform for Interactive AI Character Experiences

Sun, 10 Aug 2025 00:00:00 +0000

Immersive Conversations with Digital Einstein: Linking a Physical System and AI

Tue, 03 Dec 2024 00:00:00 +0000

EmoSpaceTime: Decoupling Emotion and Content through Contrastive Learning for Expressive 3D Speech Animation

Thu, 21 Nov 2024 00:00:00 +0000

On Multimodal Emotion Recognition for Human-Chatbot Interaction in the Wild

Mon, 04 Nov 2024 00:00:00 +0000

Chatbots With Attitude: Enhancing Chatbot Interactions Through Dynamic Personality Infusions

Mon, 08 Jul 2024 00:00:00 +0000

Multimodal Dialog Act Classification for Conversations With Digital Characters

Mon, 08 Jul 2024 00:00:00 +0000

The Personality Dimensions GPT-3 Expresses During Human-Chatbot Interactions

Sat, 01 Jun 2024 00:00:00 +0000

Artificial Intelligence for Digital Characters

Mon, 01 Jan 2024 00:00:00 +0000

Overview

This lecture provides an overview of techniques to build conversational digital characters. The main components of conversational digital characters are introduced: chatbots, speech recognition, speech synthesis, and animation synthesis. Real-life applications of such digital characters are demonstrated on different use cases (e.g., ).

Content

The lecture opens with basics on digital characters. Afterwards, the main components to build a conversational digital character are introduced. This includes the basics of natural language processing used to build a chatbot, speech recognition, speech synthesis, and animating a digital character based on motion capturing and deep learning. Further, autonomous agents based on knowledge graphs are covered. The lecture ends with real-life applications of digital characters.

Details


Level	Graduate (MSc / PhD)
Semester	Spring
Institution	ETH Zurich, D-INFK
Role	Course creator & lecturer
Years	2024 – present

Example Talk: Recent Work

Mon, 01 Jan 2024 00:00:00 +0000

Example Talk

Dr. Alex Johnson · Meta AI

Research Overview

Multimodal LLMs
Efficient training
Responsible AI

Code & Math

def score(x: int) -> int:
 return x * x

$$ E = mc^2 $$

Dual Column Layout

Left Column

Point A
Point B
Point C

Right Column

Detail 1
Detail 2
Detail 3

Main Content (2/3 width)

This column takes up twice the space of the right column.

def example():
 return "code works too"

Note
Key points in smaller column

Image + Text Layout

Results

95% accuracy
10x faster inference
Lower memory usage

Breakthrough!

Speaker Notes

Press S to open presenter view with notes!

This slide has hidden speaker notes below.

Note:

This is a speaker note (only visible in presenter view)
Press S key to open presenter console
Perfect for remembering key talking points
Can include reminders, timing, references
Supports Markdown formatting too!

Progressive Reveals

Content appears step-by-step:

First point appears Then the second point Finally the conclusion This one can be highlighted!

Note: Use fragments to control pacing and maintain audience attention. Each fragment appears on click.

Diagrams with Mermaid

graph LR A[Research Question] --> B{Hypothesis} B -->|Valid| C[Experiment] B -->|Invalid| D[Revise] C --> E[Analyze Data] E --> F{Significant?} F -->|Yes| G[Publish] F -->|No| D

Perfect for: Workflows, architectures, processes

Note: Mermaid diagrams are created from simple text. They’re version-controllable and edit anywhere!

Research Results

Model	Accuracy	Speed	Memory
Baseline	87.3%	1.0x	2GB
Ours (v1)	92.1%	1.5x	1.8GB
Ours (v2)	95.8%	2.3x	1.2GB

Key Finding: 8.5% improvement over baseline with 40% memory reduction

Note: Tables are perfect for comparative results. Markdown tables are simple and version-control friendly.

Custom Backgrounds

This slide has a blue background!

You can customize:

Background colors
Background images
Gradients
Videos (yes, really!)

Use {{< slide background-color="#hex" >}}

There’s more content below! ⬇️

Press the Down Arrow to see substeps.

Note: This demonstrates Reveal.js’s vertical slide feature. Great for optional details or deep dives.

Substep 1: Details

This is additional content in a vertical stack.

Navigate down for more, or right to skip to next topic →

Substep 2: More Details

Even more detailed information.

Press Up Arrow to go back, or Right Arrow to continue.

Citations & Quotes

“The best way to predict the future is to invent it.”

— Alan Kay

Or reference research:

Recent work by Smith et al. (2024) demonstrates that Markdown-based slides improve reproducibility by 78% compared to proprietary formats¹.

Media: YouTube Videos

Note: Embed YouTube videos with just the video ID. Perfect for demos, tutorials, or interviews.

Media: All Options

Embed various media types with simple shortcodes:

YouTube: {{< youtube VIDEO_ID >}}
Bilibili: {{< bilibili id="BV1..." >}}
Local videos: {{< video src="file.mp4" controls="yes" >}}
Audio: {{< audio src="file.mp3" >}}

Perfect for demos, interviews, tutorials, or podcasts!

Note: All media types work seamlessly in slides. Just use the appropriate shortcode.

Interactive Elements

Try these keyboard shortcuts:

→ ← : Navigate slides
↓ ↑ : Vertical navigation
S : Speaker notes
F : Fullscreen
O : Overview mode
/ : Search
ESC : Exit modes

Hidden Slide Demo (Inline Comment)

This slide is hidden using the  comment method.

Perfect for:

Speaker-only content
Backup slides
Work-in-progress content

Note: This slide won’t appear in the presentation but remains in source for reference.

Thanks

Questions?

🌐 Website:
🐦 X/Twitter:
💬 Discord:
⭐ GitHub:

All slides created with Markdown • No vendor lock-in • Edit anywhere

Note: Thank you for your attention! Feel free to reach out with questions or contributions.

🎨 Branding Your Slides

Add your identity to every slide with simple configuration!

What you can add:

Element	Position Options
Logo	top-left, top-right, bottom-left, bottom-right
Title	Same as above
Author	Same as above
Footer Text	Same + bottom-center

Edit the branding: section in your slide’s front matter (top of file).

📁 Adding Your Logo

Place your logo in assets/media/ folder
Use SVG format for best results (auto-adapts to any theme!)
Add to front matter:

branding:
 logo:
 filename: "your-logo.svg" # Must be in assets/media/
 position: "top-right"
 width: "60px"

Tip: SVGs with fill="currentColor" automatically match theme colors!

📝 Title & Author Overlays

Show presentation title and/or author on every slide:

branding:
 title:
 show: true
 position: "bottom-left"
 text: "Short Title" # Optional: override long page title

 author:
 show: true
 position: "bottom-right"

Author is auto-detected from page front matter (author: or authors:).

📄 Footer Text

Add copyright, conference name, or any persistent text:

branding:
 footer:
 text: "© 2024 Your Name · ICML 2024"
 position: "bottom-center"

Tip: Supports Markdown! Use [Link](url) for clickable links.

🔇 Hiding Branding Per-Slide

Sometimes you want a clean slide (title slides, full-screen images).

Add this comment at the start of your slide content:

<!-- no-branding -->
## My Clean Slide

Content here...

☝️ This slide uses  — notice no logo or overlays!

🔇 Selective Hiding

Hide just the header (logo + title):

<!-- no-header -->

Or just the footer (author + footer text):

<!-- no-footer -->

☝️ This slide uses  — footer still visible below!

✅ Quick Reference

Comment	Hides
`<!-- no-branding -->`	Everything (logo, title, author, footer)
`<!-- no-header -->`	Logo + Title overlay
`<!-- no-footer -->`	Author + Footer text

☝️ This slide uses  — logo still visible above!

🚀 Get Started

Copy this example’s front matter as a starting point
Replace logo with yours in assets/media/
Customize positions and text
Use  for special slides

Pro tip: Set site-wide defaults in config/_default/params.yaml under slides.branding!

Smith, J. et al. (2024). Open Science Presentations. Nature Methods. ↩︎

Experience

Tue, 24 Oct 2023 00:00:00 +0000

Personality Trait Recognition Based on Smartphone Typing Characteristics in the Wild

Sun, 01 Jan 2023 00:00:00 +0000

Seminar on Digital Humans

Thu, 01 Sep 2022 00:00:00 +0000

Overview

This seminar covers advanced topics in digital humans with a focus on the latest research results. Topics include estimating human pose and motion from images, human motion synthesis, learning-based human avatar creation, learning neural implicit representations for humans, modeling, animations, artificial intelligence for digital characters, and others. A collection of research papers is selected.

Objectives

The goal is to get an overview of actual research topics in the field of digital humans and to improve presentation and critical analysis skills.

Format

This seminar covers advanced topics in digital humans including both seminal research papers as well as the latest research results. A collection of research papers are selected covering topics such as estimating human pose and motion from images, human motion synthesis, learning-based human avatar creation, learning neural implicit representations for humans, modeling, animations, artificial intelligence for digital characters, and others. Each student presents one paper to the class and leads a discussion about the paper. All students read the papers and participate in the discussion.

Details


Level	Graduate (MSc / PhD)
Semester	Fall
Institution	ETH Zurich, D-INFK
Role	Course creator & lecturer
Years	2022 – present

Affective State Prediction from Smartphone Touch and Sensor Data in the Wild

Sat, 30 Apr 2022 00:00:00 +0000

Image Reconstruction of Tablet Front Camera Recordings in Educational Settings

Fri, 10 Jul 2020 00:00:00 +0000

Glyph-Based Visualization of Affective States

Mon, 25 May 2020 00:00:00 +0000

Affective State Prediction Based on Semi-Supervised Learning from Smartphone Touch Data

Sat, 25 Apr 2020 00:00:00 +0000

Affective State Prediction in a Mobile Setting using Wearable Biometric Sensors and Stylus

Tue, 02 Jul 2019 00:00:00 +0000

Efficient Feature Embeddings for Student Classification with Variational Auto-encoders

Sun, 25 Jun 2017 00:00:00 +0000