Eye-Tracking Through In-Ear Sensing: Cognitive-Motor Integration for Error Anticipation in High-Stakes Tasks

The imec multimodal earbud incorporates dry-electrode electrophysiology arrays, PPG, and IMU sensors and will be developed in the imec.prospect project PADRE, that will start in November 2025. Critically, its in-ear EOG electrodes will allow for the detection of eye movements—capturing gaze direction, blink frequency, fixation instability, and saccadic dynamics. These signals are currently only accessible with expensive, fragile, and cumbersome eye-tracking headsets or camera-based systems that suffer from occlusion and calibration drift. The earbud offers a compact, comfortable, and unobtrusive alternative that unlocks continuous cognitive monitoring in both controlled training setups and real-world operational contexts.

The PhD candidate will design spatio-temporal models (e.g., graph neural networks, CNN-BiLSTMs, and contrastive multimodal fusion techniques) that combine motor trajectories with electrophysiology-driven eye metrics. The aim is to predict execution errors at least 1.5 seconds in advance with ≥75% precision, distinguishing early signs of hesitation, cognitive overload, or loss of situational awareness. Experiments will be conducted across lab-based maintenance tasks, VR-based industrial simulations, and surgical training scenarios (with both novices and experts), ensuring ecological validity and cross-domain generalization.

By replacing conventional eye trackers with an unobtrusive sensing earbud, this research goes beyond state-of-the-art methods in two ways: (1) it democratizes access to eye-movement-derived cognitive markers by eliminating costly, impractical equipment, and (2) it enables true multimodal cognitive-motor fusion, where gaze dynamics are directly aligned with hand actions and physiological states. The expected outcome is a set of real-time algorithms for error anticipation, directly deployable in adaptive operator-support systems. More broadly, the project will demonstrate how next-generation wearable technologies can reshape cognitive assessment and performance monitoring in high-stakes domains.