Foundation Vision-Language models

Overview & Synopsis

In recent years, large language models (LLMs) have shown remarkable performance across diverse tasks and have begun to be widely integrated into various applications. Vision-language models (VLMs), which extend LLMs by incorporating image understanding, offer further potential. These models could reshape areas such as web development by translating webpage screenshots into code, assist in information retrieval through autonomous web navigation, and automate tasks like document classification and summarization in business contexts. However, despite the growing interest in VLMs, their development remains in its early stages, with many key aspects of the model creation pipeline lacking consensus among researchers. This thesis seeks to advance the capabilities of VLMs by both improving performance on academic benchmarks and exploring their application to real-world problems. The focus is on developing a foundation VLM, one capable of handling a wide range of tasks without requiring extensive task-specific fine-tuning. The development of large and efficient VLMs requires access to extensive, diverse, and high-quality pre-training datasets. Therefore, this thesis begins by introducing OBELICS, an open web-scale dataset of interleaved image-text documents that has undergone rigorous filtering. We describe the dataset creation process, present comprehensive filtering rules, and provide an analysis of the dataset's content. To validate OBELICS, we train an open VLM of 80 billion parameters, named Idefics1, which achieves performance competitive with proprietary state-of-the-art models at the time. Next, we investigate the architectural design of VLMs, conducting a series of experiments on pre-trained models, architecture choices, and training methods. These insights enabled the development of Idefics2, an 8-billion-parameter VLM that demonstrates state-of-the-art performance in its size class across several multimodal benchmarks at the time of its release. Finally, we offer a comprehensive overview of the most recent advances in VLM research, highlighting the strengths and weaknesses of each, addressing the major challenges in the field, and suggesting promising research directions for underexplored areas. Building on this overview, we develop Idefics3, a VLM that surpasses the performance of its predecessor, Idefics2, with particular improvements in document understanding through the creation of specialized datasets.