Learning to Perceive in Degraded Environments

Intelligent systems operating in underwater environments, adverse weather, darkness, smoke, cluttered scenes, or under partial sensor failure must continuously determine not only what they can perceive, but also how they should acquire information in order to perceive effectively. This need to adapt how information is acquired and processed extends also across autonomous systems, industrial monitoring, automotive perception, and intelligent sensing platforms operating in dynamic real-world environments. In such settings, the value of information is highly context-dependent: the usefulness of a sensing modality, sensing configuration, or computational strategy may change over time, across locations, and according to the mission/task being performed. Existing research has made significant advances in robust perception, multimodal fusion, active sensing, and resource-aware AI. However, these areas have largely evolved independently, leaving a fundamental open question: which properties must a learned representation capture in order to predict the future usefulness, reliability, and cost of information under changing environmental and operational conditions, and thereby support improved perception decisions?

This PhD investigates the learning of effective and efficient representations for active multimodal perception in degraded and uncertain environments. The central hypothesis is that perception systems require an intermediate representation layer that captures not only what is observed, but also the latent factors that determine the expected value, reliability, and cost of future observations. Such representations should enable perception systems to estimate these quantities as environmental conditions and mission priorities evolve. This information can subsequently support decisions such as selecting sensing modalities, adapting sensing strategies, or allocating computational resources.

A key challenge is that these representations must simultaneously support multiple and often competing objectives. They must be informative enough to improve perception performance, compact enough to operate under constraints such as latency, energy consumption, bandwidth, battery life, and computational budgets, and robust across a range of sensing conditions and mission objectives. The research therefore focuses on learning representations that balance effectiveness and efficiency while remaining actionable for decision-making in resource-constrained systems.

An additional objective is to ensure that the learned representations expose the factors that underlie perception decisions (i.e., estimates of uncertainty, expected benefit, and resource implications), providing a deliberate layer of interpretable communication between perception and decision-making processes. Rather than treating perception decisions as opaque outputs, the research investigates how systems can communicate, in human-understandable terms, the trade-offs underlying their sensing choices. Such capabilities are increasingly important in safety-critical and long-duration deployments where operators must understand, trust, and audit autonomous behaviour.

Thus, this PhD research should be naturally structured around three hierarchical pillars:

• Learning representations of information value: The primary focus is to understand which properties a learned representation must capture in order to predict the reliability and cost of information under changing environmental conditions, mission objectives, and resource constraints. This research explores new directions in representation learning, uncertainty modelling, and multimodal AI, supported by large-scale experiments in realistic simulation and physics-based digital worlds.
• Guiding active multimodal perception: Building on these representations, the research investigates how they can support sensing and computation decisions, enabling systems to adapt how information is acquired and processed in response to uncertainty, operational context, and available resources. The resulting algorithms will be trained, tested, and demonstrated in realistic digital worlds combining robotics, physics-based rendering, and multimodal sensor simulation, before being transferred to real-world experimental platforms.
• Bridging perception and auditable AI reasoning: Finally, the research explores how learned representations can become a semantic interface between sensing systems and next-generation AI models, allowing autonomous agents to reason about uncertainty, information value, and resource costs when deciding how to perceive and act.

The resulting scientific framework aims to provide a principled basis for learning representations that guide perception decisions under uncertainty and resource constraints in a principled, auditable way. By linking representation learning with active multimodal perception, the research seeks to advance the development of more adaptive and trustworthy sensing systems that respond effectively to the complexity of real-world environments. This is a strong outcome towards the broad adoption of autonomous, intelligent systems and should actively contribute to the maturity and scalability of imec’s next generation sensing and perception systems.

Our ideal candidate for this position has the following skills:

• You have a Master’s degree in AI, Computer Science, Electrical Engineering, Robotics, Applied Mathematics, or a related field.
• You have solid knowledge of machine learning and deep learning, with an interest in representation learning, multimodal perception, and embedded or resource-aware AI.
• You are interested in designing algorithms that enable intelligent systems to adapt how they acquire and process sensor information.
• You have good programming skills and are flexible in using software tools and machine-learning frameworks such as Git, PyTorch, or TensorFlow.
• Experience with multimodal sensor data, sensor fusion, active sensing, robotics, or autonomous systems is considered a plus.
• Understanding the properties and limitations of different sensor modalities is considered a plus.
• Experience with uncertainty estimation or resource-constrained machine learning is considered a plus.
• You can plan and carry out research tasks independently while contributing effectively to a multidisciplinary team.
• You have strong analytical skills and can critically interpret, communicate, and document research results.
• You are responsible, curious, communicative, and flexible, with a willingness to work across application domains.
• Your English is fluent, both spoken and written.