One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Image (Short Paper)

Large multimodal models (LMMs) enable a wide range of zero-shot capabilities spanning a variety of input modalities, such as language, vision, and audio. However, similar to unimodal large language models (LLMs), they can suffer from hallucinations, generating outputs that are not grounded in factual information. Retrieval augmented generation (RAG) attempts to address this issue by retrieving trusted information chunks, that are relevant to the user query, directly from a knowledge base (KB). The retrieved information is then included in the input to the generative model to inform its generation.

The effectiveness of RAG relies primarily on the trustworthiness of the information in the KB. Challenging this assumption, recent work has shown that textual RAG pipelines are vulnerable to poisoning attacks, where an attacker injects malicious information into the KB. To create an impactful attack, the injected information must simultaneously (1) have a high chance of being retrieved, and (2) influence the output of the generative model. These attacks have recently been extended to multimodal RAG (M-RAG) pipelines by concurrent research.

Both of those concurrent works consider that the user queries and the KB consist of text-image pairs, thus the text modality is common between queries, KB items, and LMM outputs. This shared modality greatly simplifies the attack procedure.

Therefore, it remains unclear whether M-RAG poisoning would still be possible in this practical setting. In this paper, we aim to bridge this gap in the literature by investigating poisoning attacks on visual document RAG (VD-RAG). We adopt a gradient-based adversarial approach to craft an image that is retrieved, and consistently influences the generation of every possible user query, thus leading to a universal denial-of-service (DoS) attack against the RAG pipeline.

A visual document RAG pipeline consists of three main components. First, a knowledge base (KB) K = {I₁, …, I_K} containing a set of K images, each corresponding to a page in a document. Second, a retriever R uses a multimodal embedding model E(.) that projects user queries (text) and KB images into a common vector space. The retriever then computes a similarity score S(E(q), E(I)) between a user query q and each image in I ∈ K. Common similarity metrics for RAG retrievers include cosine similarity and MaxSim. For each user query q, the retriever retrieves the top-k relevant items from K according to the similarity score, where k ≪ |K|. In other words, the retriever computes R(q, K) = top-k_I∈K S(E(q), E(I)).

The third component is a generator G which is a vision language model (VLM) used to generate a response g to the user’s query q with the retrieved images in its context window. That is, g = G(q, R(q, K)).

We consider an attacker that aims to disrupt the operation of the visual document RAG system by causing the retriever to retrieve irrelevant images and the generator to generate unhelpful responses to user queries. To achieve this goal, the attacker is capable of injecting documents/images into the KB. In our work, we assume a weak attacker that can inject only one malicious image I′ into the KB s.t. K′ = K ∪ I′. In order for this malicious image to cause disruption to the RAG pipeline, this image has to be crafted to be retrieved often, and when retrieved, it has to cause the VLM to generate unhelpful responses.

We examine both white-box and black-box settings.

A successful RAG poisoning attack must meet two conditions (modified for our goal of a universal untargeted attack).

In order to compute a malicious image I′ that meets the above two conditions, we adopt the framework of gradient-based adversarial examples, initially proposed against neural network based image classifiers. In particular we jointly optimize the image to minimize a loss function L_RAG capturing both a retrieval loss L_R and a generation loss L_G as follows

where λ_R, λ_G are coefficients,

where Q is the set of user queries used to train the attack, CE(.) is the cross entropy loss, a* is the desired target answer by the attacker

We evaluate our attacks on two visual document retrieval datasets taken from the ViDoRe benchmark versions 1 and 2. We use a mix of embedding models that range in their sizes, recency, and target applications.

We split each dataset into a training set (80%, used to optimize the malicious image) and a test set (20%). For the above models, we assume the retriever only retrieves the top-1 relevant image from the KB.

We evaluate the RAG system and the attack using the following performance metrics:

We observe that attacks can almost perfectly satisfy both the retrieval and the generation conditions starting from α = 8/255. Therefore, in the rest of the paper, we consider only attacks with α = 8/255. Despite the success of the attack, the perturbed image is almost indistinguishable from the original.

As the previous results indicate, we observe high values of ASR-R, ASR-G-Hard, and ASR-G-Soft for most of the considered models, except ColPali-v1.3 which is almost unaffected by our attack. Increasing the allowed perturbation to α = 128/255 and training over 1500 gradient steps did not produce better results against ColPali-v1.3. Otherwise, CLIP-ViT-LARGE and Jina-CLIP-v2 almost always retrieve our malicious image.

Regarding generation, our attack consistently causes SmolVLM-Instruct and Qwen2.5-VL-3B-Instruct to generate the target answer verbatim for almost all user queries in the test dataset. We also observe an interesting interaction where Jina-CLIP-v2 seems to somewhat inhibit the learning of effective attacks against generation. In general, the results show that our attacks are successful against a broad range of models.

While the above evaluations assumed white-box access to the attacker, we also evaluated our attacks in the black-box setting, where the adversary does not have access to the embedding model and VLM used in the RAG system. We did not observe any transferability of the attack across the different embedding and VLM models. Attacks trained against a certain combination (embedding model, VLM) only work against the same combination. Therefore, exploring attacks that are trained simultaneously against multiple models is a direction for future research.

• We do not consider defenses to our proposed attack; however, we plan to evaluate our attack against a variety of defenses including re-rankers, multimodal LLM-as-a-Judge, and poisoned sample detectors in future work. Additionally, we aim to examine whether adaptive attacks could be crafted against specific defenses.
• Our attacks can be detected by filtering images that are retrieved too frequently; however, our attack uncovers a fundamental weakness of M-RAG pipelines that could be useful for hardening through adversarial training.
• Our evaluation is not comprehensive, as we do not consider a large number of embedding models, VLMs, datasets, metrics, and transferability evaluations. In future work, we aim to provide an extensive evaluation of the attack showing the generalization of the results observed in this paper.

In this paper, we demonstrated the first poisoning attack against VD-RAG systems. Our attack showcased that conventional embedding models and VLMs are vulnerable to adversarial perturbation, where a single injected image is capable of causing a DoS of the entire RAG pipeline (both retrieval and generation). We also observed notable adversarial robustness of the ColPali-v1.3 embedding model and provided hints into why this is the case. Nevertheless, our attack provides the first step towards understanding the vulnerability of multimodal RAG systems and enhancing their robustness for future adoption.