Hardware-Aware Data Attribution Methods for Foundation Models

Foundation models (large language models, LLMs, and vision–language models, VLMs) increasingly power high-stakes applications, yet their decisions and outputs remain difficult to trace back to the data and mechanisms that generated them—complicating copyright and license compliance as well as trust and accountability. This PhD will design scalable, faithful, and hardware-efficient attribution methods that work end-to-end—from training data to model outputs—while being deployable on contemporary accelerators (GPU/TPU/NPU) and edge platforms; by linking model behavior to specific data sources, the project enables provenance auditing and responsible data sharing. 

Methodologically, you will build on training-data influence estimation (e.g., influence-function and gradient-tracing families) with approximations that avoid expensive second-order computations and make them tractable for billion-parameter models under strict compute and memory budgets. You will extend unimodal attribution to multimodal settings (image–text, video–text) to quantify cross-modal contributions in VLMs such as CLIP-style architectures. Beyond scores, the project will pair attributions with Natural Language Explanations (NLEs) so users receive concise, concept-level rationales rather than raw heatmaps. 

A central focus is the interaction with hardware: designing attribution pipelines that are co-optimized with accelerator characteristics (memory hierarchy, bandwidth, mixed precision, sparsity), enabling fast batched influence queries. You will prototype kernels and scheduling strategies that keep attribution within tight latency/energy envelopes without sacrificing faithfulness. 

Finally, you will contribute rigorous evaluation and consider applications in domains from media to healthcare to demonstrate compliance benefits alongside technical performance. The work leverages and extends prior IMEC-VUB-IMS results on multimodal explanations (e.g., Sammani & Deligiannis. Zero-Shot Natural Language Explanations. ICLR 2025).