Embodied Conversational Agents | Dr. Rafael Wampfler

Digital Einstein

Tue, 01 Jun 2021 00:00:00 +0000

Overview

Digital Einstein is a flagship embodied conversational agent that brings the historical figure of Albert Einstein to life through real-time multimodal AI interaction. The system combines speech recognition and synthesis, facial animation, gesture control, and a cognitively grounded language understanding pipeline to deliver immersive, personality-consistent conversations.

Digital Einstein serves three interconnected roles:

Research platform — a testbed for studying human–agent interaction in constrained embodied settings, yielding insights on affective computing, personality modeling, dialog act classification, and conversational AI.
Education platform — a live demonstration of conversational AI and multimodal deep learning deployed in university events, science outreach, and public engagement.
Public engagement tool — reaching thousands of visitors globally at scientific conferences, tech summits, museums, and public events, generating sustained international recognition for ETH Zurich.

Motivation

How can AI systems convincingly portray a well-known historical personality — someone whose knowledge, values, and speaking style are culturally established — in real-time dialogue with arbitrary members of the public? This challenge crystallizes core problems in interactive AI: maintaining factual and characterological consistency, adapting dynamically to unpredictable user inputs, and delivering a compelling embodied experience at scale.

Digital Einstein was conceived as both a scientific challenge and a communication vehicle: making abstract advances in AI tangible for general audiences while simultaneously driving rigorous research on the underlying problems.

Approach

The system is built on a full-pipeline architecture described in the SIGGRAPH 2025 paper “A Platform for Interactive AI Character Experiences”. Key components include:

Perception layer: Real-time speech recognition via Microsoft Azure Speech Services and multimodal input processing through a webcam-based vision pipeline, including face detection, user characterization, head pose estimation, and re-identification.
Cognitive reasoning layer: Knowledge-grounded dialogue management with integrated response generation, powered by GPT-4.1 mini, featuring dynamic personality infusion that adapts outputs to user-selectable archetypes: Digital Einstein, Rude Bulldozer, Drama Volcano, Zen Master, and Hashtag Prophet.
Animation synthesis: Data-driven facial animation synchronized with speech output using NVIDIA Audio2Face, blended with emotion-conditioned expressions, and complemented by a curated library of motion-captured body animations categorized by avatar state.
Embodiment: A stylized Albert Einstein avatar rendered in Unity on a 65-inch display, integrated into a themed early-20th-century physical environment with spatial audio, a hidden microphone, and physical personality sliders built from potentiometers and an Arduino.

The SIGGRAPH Asia 2024 demonstration paper “Immersive Conversations with Digital Einstein: Linking a Physical System and AI” details the physical installation setup, including the integration of an animatronic head with the real-time AI pipeline at the Tokyo venue.

Key Results

Digital Einstein has been demonstrated at over 20 major events worldwide, including:

SIGGRAPH Asia 2024 (Tokyo, Japan) — Emerging Technologies
SIGGRAPH 2025 (Vancouver, Canada)
GITEX Global 2024 & 2025 (Dubai, UAE) — Swiss Pavilion
World Economic Forum 2024 & 2026 (Davos, Switzerland)
Berlin Science Week 2025 (Berlin, Germany)
Swiss Re Resilience Summit 2024 (Rüschlikon, Switzerland)
Microsoft Initiative to Advance AI Diffusion in Switzerland 2025 (Berne)
After the Algorithm Festival 2026 (Zurich, Switzerland)

The project has generated sustained international media coverage and public interest, positioning ETH Zurich as a world leader in embodied conversational AI.

Swiss Ambassador to the UAE, Arthur Mattli, interacting with Digital Einstein at GITEX Global in Dubai.

Learn More

Primary Publications

R. Wampfler, C. Yang, D. Elste, N. Kovačević, P. Witzig and M. Gross (2025). A Platform for Interactive AI Character Experiences. Proceedings of the SIGGRAPH Conference Papers ‘25 (Vancouver, Canada, August 10–14, 2025), pp. 1–11.

R. Wampfler, N. Kovačević, P. Witzig, C. Yang, M. Gross (2024). Immersive Conversations with Digital Einstein: Linking a Physical System and AI. In SIGGRAPH Asia 2024 Emerging Technologies (SA ‘24) (Tokyo, Japan, December 3–6, 2024).

Virtual Psychotherapist

Sun, 01 Jan 2023 00:00:00 +0000

Overview

Access to evidence-based psychotherapy remains severely limited worldwide — constrained by long waitlists, resource scarcity, and geographic disparities. The Virtual Psychotherapist project develops embodied conversational AI agents that complement clinical care by extending access, supporting therapeutic practice, and enabling scalable training.

This initiative is conducted in close collaboration with the University of Lucerne, providing clinical expertise and direct access to real therapy data.

Motivation

Two distinct but complementary challenges motivate this project:

Scaling patient access: Many individuals who would benefit from psychotherapy cannot access it due to cost, waitlists, or geography. AI-based companions that operate between sessions, provide continuous support, and conduct structured therapeutic conversations could meaningfully improve outcomes at scale.
Improving therapist training: Training clinicians in evidence-based interventions requires repeated practice with feedback — but opportunities for safe, standardized training are limited by ethical and resource constraints. Simulated patient systems powered by AI can provide unlimited deliberate practice in controlled settings.

System Architecture

Both applications are built on a shared platform combining:

Large language model-based dialogue: State-of-the-art LLMs for contextually appropriate, therapeutically grounded response generation
Retrieval-augmented generation (RAG): Responses grounded in evidence-based therapeutic literature, reducing hallucinations and ensuring adherence to clinical frameworks
Real-time psychological analysis: Parallel processing pipelines that extract facts, detect psychological flexibility processes, recognize emotions, and monitor safety in real time
Embodied avatar presentation: Synchronized speech synthesis and 3D avatar animation delivered through mobile and desktop applications
Clinician oversight: Structured analysis outputs accessible to supervising therapists, enabling human-in-the-loop clinical governance

Patient-Facing Application

The patient-facing component enables individuals to conduct therapeutic conversations with an embodied avatar between sessions. The system follows process-based therapy principles — particularly Acceptance and Commitment Therapy (ACT), an empirically supported approach targeting psychological flexibility through six core processes: acceptance, cognitive defusion, present-moment awareness, self-as-context, values clarification, and committed action.

Critically, the system is designed for clinical supervision, not autonomous intervention. All session data is structured and accessible to the supervising therapist, who can monitor progress and intervene as needed.

An evaluation against responses from professional psychotherapists demonstrated that the system’s responses were rated significantly higher on understanding, interpersonal effectiveness, collaboration, and ACT alignment — while emphasizing that clinical judgment and the therapeutic relationship remain irreplaceable.

Therapist Training Application

The training application enables psychotherapists to practice and refine therapeutic techniques through role-play interactions with a simulated patient. Key features include:

Clinically grounded patient simulation: Virtual patient behavior conditioned on profiles derived from real therapy sessions, covering a range of clinical presentations and scenarios (suicidality, resistance, heightened anxiety, therapeutic rupture, and more)
Real-time ACT fidelity feedback: An automated evaluator assesses each therapist utterance for adherence to ACT principles, providing immediate visual feedback and the option to retry alternative responses
Configurable scenarios: Therapists can select specific clinical scenarios to target their practice

A systematic evaluation across 49 therapy transcripts identified GPT-4o-mini as the optimal feedback model, achieving the closest alignment with human supervisor ACT fidelity ratings.

Safety and Ethics

Safety is a primary design constraint. The system includes:

Crisis detection: Explicit classification of suicidal ideation and self-harm signals, triggering immediate presentation of crisis resources and mandatory clinician review
Unsafe-interaction detection: Identification of conditions (e.g., active psychosis, mania) where LLM interaction may be counterproductive, with protocol-defined fallback responses
Non-autonomous design: The system is explicitly positioned as a complement to clinical care, not a replacement — structured to require and facilitate clinician oversight

Key Results

Patient-facing system responses rated significantly higher than human therapist responses by automated evaluation and expert psychotherapists across understanding, collaboration, and ACT alignment
Therapist training simulation rated as realistic by practicing psychologists; turn-by-turn feedback shown to increase therapist awareness of intervention choices
Automated ACT fidelity assessment achieves strong agreement with human expert ratings across 49 therapy transcripts

Research Partners

University of Lucerne

RehaBot

Mon, 01 Jan 2024 00:00:00 +0000

Overview

RehaBot is an embodied conversational agent designed to support patients in rehabilitation and home-care settings. The avatar represents a medical professional — capable of conducting structured patient interactions, administering assessments, and delivering health education — to help bridge the gap between in-clinic care and independent recovery at home.

This project is developed in collaboration with Inselspital Bern (University Hospital) and Bern University of Applied Sciences (BFH).

Motivation

Rehabilitation after medical treatment — whether from stroke, orthopedic surgery, cardiac events, or chronic disease — requires sustained patient engagement over weeks or months. Yet contact with healthcare professionals is necessarily episodic, leaving long gaps during which patients must self-manage. Lack of guidance, motivation, and timely feedback during these intervals is a major driver of poor rehabilitation outcomes and preventable hospital readmissions.

An embodied conversational agent that patients can interact with at home — to receive reminders, answer questions, conduct structured assessments, and provide health education — addresses this gap directly. By combining medical knowledge with empathetic communication and a human-like embodied presence, RehaBot aims to make professional-quality support continuously available between clinical appointments.

Approach

RehaBot integrates several complementary AI capabilities within a unified embodied avatar system:

Medical knowledge integration: Structured clinical knowledge relevant to the patient’s rehabilitation pathway, enabling accurate and safe responses to health questions
Conversational assessment: The ability to administer structured health questionnaires and functional assessments through natural spoken dialogue, adapting pacing and clarification to individual patient needs
Empathetic communication: Affective modeling that allows the agent to detect and respond to emotional signals in patient speech — frustration, discouragement, anxiety — with appropriate supportive responses
Health education: Accessible explanations of rehabilitation exercises, medication adherence, warning signs, and self-management strategies, adapted to the patient’s comprehension level
Patient-professional interface: Structured summaries of patient interactions accessible to supervising clinicians, supporting continuity of care and early detection of clinical deterioration

The system is built on the same core platform as the Digital Einstein project, enabling rapid deployment of new capabilities while maintaining consistent embodied presentation quality.

Key Results

Recent work exploring embodied conversational interfaces for personal health data reflection demonstrates that users who engage with health information through a conversational agent formulate significantly more specific and actionable health plans compared to traditional dashboard-based exploration. Embodied conversation lowers the cognitive burden of interpreting health data and supports a shift from passive data inspection to active health sensemaking.

Research Partners

Inselspital Bern (University Hospital of Bern)
Bern University of Applied Sciences (BFH)

MIND — Cognitive Health Monitoring

Wed, 01 Jan 2025 00:00:00 +0000

Overview

The MIND (Monitoring and Supporting Cognitive Health) project creates a mobile platform for the early detection of cognitive impairment in aging populations. Early identification of decline — before it significantly impacts daily life — enables timely intervention, supports independent living, and improves long-term outcomes for older adults and their families.

MIND is part of the Future Health Technologies 2 (FHT2) initiative, a major research program funded by the National Research Foundation Singapore (NRF) CREATE grant.

Motivation

Cognitive decline — from mild cognitive impairment to dementia — affects hundreds of millions of people worldwide. Current clinical detection relies predominantly on periodic in-clinic assessments that are episodic, costly, and often catch decline only after substantial impairment has occurred. A continuously operating, passive monitoring system that can detect subtle early warning signs in everyday behavior could transform the standard of care.

Two key insights motivate the MIND approach:

Movement and navigation encode cognition: Changes in how people navigate their environment — route choices, wayfinding strategies, spatial memory — are among the earliest and most sensitive indicators of cognitive decline. Analyzing GPS and sensor traces at scale can reveal these changes.
Language reflects cognitive health: Subtle changes in vocabulary, sentence complexity, topic management, and response latency in everyday conversation are measurable markers of cognitive change, detectable through automated language analysis.

Approach

MIND integrates three complementary AI systems into a unified mobile platform:

Large Geospatial Models (LGMs)

Large Geospatial Models analyze longitudinal GPS and mobility traces to detect patterns indicative of cognitive change. These include:

Route repetitiveness: Increasing restriction of daily movement range
Navigation errors: Unusual detours or disorientation in familiar environments
Mobility diversity: Changes in the variety of visited locations over time

LGMs are trained on large-scale mobility datasets and fine-tuned to identify individual-level deviations from baseline behavior.

LLMs for Conversational Cognitive Assessment

Large language models analyze conversation transcripts from daily interactions with the MIND app, detecting cognitive markers such as:

Reduced lexical diversity and increased use of filler words
Difficulty with topic maintenance and coherence
Slowed response generation and increased hesitation
Confusion or confabulation in response to everyday questions

These markers are tracked longitudinally to identify meaningful change relative to the individual’s own baseline.

Embodied Avatars for Guided Assessment and Training

An embodied conversational avatar conducts structured cognitive assessments — such as memory recall tasks, verbal fluency tests, and orientation questions — through natural spoken dialogue. The avatar also guides cognitive training exercises designed to maintain cognitive reserve.

The embodied format is critical: older adults are more likely to engage regularly with an interactive, socially present agent than with a text-based questionnaire or passive sensor. The avatar adapts its communication style to individual users, adjusting vocabulary, pacing, and support level to ensure accessibility.

Integration and Privacy

All three systems operate on a shared mobile platform with strong privacy protections. Data is analyzed on-device where possible, and users maintain full control over what is shared with care providers. The platform produces structured reports for geriatricians and primary care physicians, enabling early clinical intervention.

Significance

MIND represents a convergence of several research threads — geospatial AI, conversational AI, affective computing, and embodied interaction — applied to one of the most pressing health challenges of our time. By enabling early, passive, continuous monitoring, the platform aims to support successful aging in place and to delay or prevent the transition to care dependency.

Research Partners and Funding

National Research Foundation Singapore (NRF) CREATE — Future Health Technologies 2 (FHT2)
Bond University (Australia)
Collaboration with life sciences and medicine partners in Singapore

Talking to Your Data: Exploring Embodied Conversation as an Interface for Personal Health Reflection

Mon, 23 Mar 2026 00:00:00 +0000

A Platform for Interactive AI Character Experiences

Sun, 10 Aug 2025 00:00:00 +0000

Immersive Conversations with Digital Einstein: Linking a Physical System and AI

Tue, 03 Dec 2024 00:00:00 +0000