Lightweight Adaptation of Large Language and Vision Models in Robotics

The rapid proliferation of large-scale multimodal foundation models, combining state-of-the-art language and vision representations, has ushered in unprecedented capabilities for perception, reasoning, and decision-making in complex environments. These models, commonly referred to as multimodal large language and vision models (LLVMs), are pretrained on massive heterogeneous datasets and demonstrate remarkable generalization across a variety of downstream tasks. However, their sheer scale-often encompassing billions of parameters-poses formidable challenges for deployment in resource-constrained and real-time systems, such as autonomous robots, where memory, computational throughput, and energy budgets are tightly limited. Parameter-efficient fine-tuning (PEFT) has emerged as a pivotal paradigm to address this bottleneck, enabling task-specific adaptation by modifying only a small subset of model parameters, or by introducing compact auxiliary modules, while leaving the majority of the pretrained weights frozen. By doing so, PEFT maintains the extensive knowledge encoded in the foundational model while drastically reducing both computational and storage overheads, facilitating rapid adaptation to new robotic tasks, environments, and user instructions. This review provides a comprehensive survey of the state-of-the-art PEFT methodologies, including low-rank adaptation (LoRA), adapter modules, prompt tuning, prefix tuning, and hybrid approaches, with a focus on their application to multimodal LLVMs in robotics. We explore the mathematical underpinnings of these techniques, formalizing their parameter constraints, forward-pass computations, and optimization strategies, and we illustrate their integration into robotic perceptionto-action pipelines using schematics and comparative tables. Special emphasis is placed on applications across vision-language navigation, robotic manipulation, and human-robot interaction, highlighting how PEFT facilitates few-shot learning, continual adaptation, and multi-modal reasoning without the need for full-model retraining. Furthermore, we discuss critical challenges, including distributional shifts between pretraining and operational domains, real-time inference constraints, interpretability, safety, and verification, while outlining promising future directions such as hierarchical modular adaptation, online learning, and embodied pretraining to bridge these gaps. Overall, PEFT provides a principled and practical framework for harnessing the immense capabilities of multimodal LLVMs in robotics, enabling autonomous systems that are adaptable, efficient, and capable of integrating complex multimodal knowledge into actionable, real-world behavior.

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.