One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Image

Retrieval-augmented generation (RAG) has recently gained significant attention in both research and practical large language model (LLM) deployments. RAG augments the parametric knowledge of LLMs by retrieving relevant chunks of information from external knowledge bases (KBs), thus improving groundedness and reducing hallucinations. One of the most common sources of external knowledge is PDF documents (e.g., user manuals, health records, academic articles). Most RAG pipelines for PDFs either extract only the main text and ignore images, charts, and tables, or apply optical character recognition (OCR) to extract text from those visual elements.

Recently, Faysse et al. (2024) were first to establish the promise of visual document retrieval (VDR) by regarding each page in a PDF document as an image and leveraging the recent breakthroughs in multi-modal embeddings and vision language models (VLMs). The same approach has been applied to multi-page document understanding and RAG, leading to visual document RAG (VD-RAG) pipelines that show significant performance improvements compared to textual RAG pipelines. VD-RAG is also used in practical settings, for example, Colette, a self-hosted VD-RAG product that can interact with technical documents.

The effectiveness of RAG relies primarily on the trustworthiness of the information in the KB. Challenging this assumption, recent work has shown that existing RAG pipelines are vulnerable to poisoning attacks, where an attacker injects malicious information into the KB. To create an impactful attack, the injected information must simultaneously (1) have a high chance of being retrieved, and (2) influence the output of the generative model. However, the extent to which KB poisoning can disrupt VD-RAG pipelines has not yet been explored in the literature.

In this paper, we bridge this gap by investigating the vulnerability of VD-RAG to poisoning attacks. We consider the white-box attack setting by adapting projected gradient descent (PGD) with a multi-objective loss, which we refer to as MO-PGD, to balance the optimization of the retrieval and generation objectives when crafting the malicious image. First, we propose a stealthy targeted attack objective where the image only influences specific queries, thus causing targeted disinformation on a certain topic. Second, we propose a universal attack objective where the image is optimized to be retrieved and influence generation for all queries, thus causing a denial-of-service (DoS) attack against the VD-RAG pipeline. Furthermore, we consider three black-box attack variants for both the targeted and universal objectives: (1) leveraging existing multi-modal generative models, (2) exploiting direct transferability across VD-RAG pipelines, and (3) optimizing the image over an ensemble of candidate embedding models and VLMs.

Zhuang et al. (2025) investigated the vulnerability of document retrievers to adversarial attacks; however, the work did not consider the joint problem of retrieval and generation.

Focusing on multi-modal RAG systems (image-text pairs), Ha et al. (2025) proposed targeted and universal poisoning attacks under both the white-box and black-box settings. Shortly after, Liu et al. (2025) proposed a targeted disinformation poisoning attack against multi-modal RAG systems. Nonetheless, the above two works consider KBs including image-text pairs, with the text modality greatly simplifying the attacks (e.g., including the targeted user query and/or malicious answer verbatim in the injected text). Importantly, all the above works attack outdated multi-modal embedding models (e.g., CLIP) that include known vulnerabilities, such as the so-called modality gap.

A VD-RAG pipeline consists of three main components. The first is a knowledge base K = {I₁, …, I_K} containing a set of K images, each corresponding to a page in a document. The second is a retriever R which uses a multi-modal embedding model E(·) that projects user queries (text) and KB images into a common vector space. The retriever then computes a similarity score S(E(q), E(I)) between a user query q and each image in I ∈ K. Common similarity metrics for RAG retrievers include cosine similarity and MaxSim proposed by Faysse et al. (2024). For each user query q, the retriever retrieves the top-k relevant images from K according to the similarity score, where k ≪ |K|. The third component is a generator G, which is a VLM that generates a response g to the user’s query q with the retrieved images in its context window. That is, g = G(q, R(q, K)).

We consider an attacker that aims to disrupt the operation of the VD-RAG system by causing the retriever to retrieve a malicious adversarial image and the generator to output unhelpful responses to user queries.

In our work, we assume a weak attacker that can only inject one malicious image I′ into the KB, such that K′ = K ∪ I′, as a single image is sufficient to demonstrate the vulnerability of VD-RAG. Scaling to multiple images would amplify the impact, however, it would reduce the stealthiness of the attack.

Building upon the work of Zou et al. (2024), a successful RAG poisoning attack must meet two conditions.

To compute a malicious image I′ that meets the above two conditions, we adopt a gradient-based adversarial example framework, initially proposed against neural network-based image classifiers. In particular, we extend the widely-used PGD optimization algorithm to jointly optimize the image to minimize a multi-objective loss function L_RAG capturing both a retrieval loss L_R and a generation loss L_G as follows:

where λ_R, λ_G are attacker-chosen coefficients controlling the relative weights of the two adversarial objectives. The adversary chooses a subset of positive target queries Q⁺ ⊆ Q that it wishes to influence; the remaining queries Q⁻ = Q \ Q⁺ are termed negative queries. The set of answers A is also divided into malicious answers a⁺_i desired by the attacker for targeted queries q⁺_i ∈ Q⁺ and benign ground truth answers a⁻_i for q⁻_i ∈ Q⁻.

The retrieval and generation losses are defined as follows:

where S(·) is the similarity measure between the query and the image embeddings, CE(·) is the cross entropy loss, I_k−1 is a randomly sampled subset, with cardinality |I_k−1| = k − 1, of the attacker-owned KB image dataset I. Note that I_k−1 ∪ I′ represents the top-k images retrieved simulated by the attacker.

To minimize the loss L_RAG we adopt a multi-objective variant of PGD, referred to as MO-PGD, which iteratively updates the adversarial image I′ using the following formula:

where t ∈ {1, …, T} is the iteration index, ϵ is the perturbation budget controlling the attack stealthiness, α is the learning rate, ∇ is the gradient operator, I′₀ is a benign image, and I′_T is the final adversarial image to be injected to the KB.

The injected image is used to realize one of two malicious objectives:

1. a targeted attack where the image should only be retrieved and influence generation for a specific query (|Q⁺| = 1, Setting I), or a subset of queries (|Q⁺| ≪ Q) with either a single target answer (∀q⁺_i, q⁺_j ∈ Q⁺, a⁺_i = a⁺_j, Setting II) or distinct target answers (∀q⁺_i, q⁺_j ∈ Q⁺, a⁺_i ≠ a⁺_j, Setting III);
2. a universal attack where the injected image should both be retrieved and influence generation for any possible user query (Q⁺ = Q).

The first objective corresponds to stealthy and specific attacks, such as spreading disinformation on specific topics; the second objective corresponds to a DoS attack against the availability of the VD-RAG system.

We examine different levels of attacker knowledge through variants of the previous attack definition, varying from full white-box access to the black-box setting. In the black-box setting the attacker does not know the target models. We investigate three attack variants at increasing levels of difficulty for crafting malicious images: a Prompt-based Attack, a Direct Transfer Attack, and a Model Ensemble Attack.

We evaluate the attacks on two visual document retrieval datasets taken from the ViDoRe benchmark versions 1 and 2. We use a mix of embedding models that range in size, recency, and target applications.

We split the queries of each dataset into a set used to optimize the malicious image (80%) and a set to evaluate the attack for the universal objective (20%). For all the above models, unless otherwise stated, we assume the retriever only retrieves the top-1 relevant image from the KB.

We evaluate the RAG system and each attack using the following performance metrics for retrieval, where ↑ / ↓ denote their relation to the performance of the attacker:

And the following performance metrics for generation:

For all evaluation metrics, the superscript @k signifies that the metric is reported when the top-k relevant images are retrieved. Furthermore, we denote by @−1 that we force the malicious image to be retrieved as the only context image, which decouples the generation performance from retrieval, so we can report on the performance of each component of the attack individually. Similarity metrics are computed using the Jina-Embeddings-V3 text embedding model. We use an embedding model different from those employed by the VD-RAG system, and the attacks, to exclude the possibility of bias.

We observe that attacks can almost perfectly satisfy both the retrieval and the generation conditions starting from α = 8/255. Therefore, in the rest of the paper, we consider only attacks with α = 8/255. Despite the success of the attack, the perturbed image is almost indistinguishable from the original.

We empirically evaluate the vulnerability of VD-RAG to the targeted attack across three settings, with increasing number of targeted queries and answers: I) One Query, II) Multiple Queries, and III) Multiple Queries & Answers. Across all settings, the malicious image is never retrieved for unrelated queries, so the false positive rate (FPR) is always 0.

In this setting, we target a single query in the dataset such that the malicious KB entry is retrieved and it induces the VLM to generate a specified malicious answer generated by GPT-4o.

Focusing on the white-box setting, the results show that an attacker with full knowledge of the target models can succeed in compromising the RAG system. For retrieval, the malicious image is always retrieved as the most relevant image for the target query when CLIP-L is the embedding model. For more sophisticated embedding models (i.e., ColPali and GME), the malicious image is almost always retrieved within the top-5 most relevant images. For generation, the generated answer is semantically similar to the target answer (ASR-G_Sim@−1 ≥ 0.8) for most model combinations. Note that ASR-G_Sim@−1 is relatively lower when ColPali or GME is used, as the attack optimization is not able to balance the retrieval and generation objectives. Further note that the malicious image has high specificity and does not influence the generated answers for untargeted queries, even when retrieved, as shown by the SIM-G_Neg@−1 values.

Regarding the black-box attack variants, we observe no transferability between the models when applying the Direct Transfer Attack. The same applies for the Out-set Model Ensemble Attack (i.e., when the model ensemble does not include the actual models used). However, when the model ensemble includes the models employed in the VD-RAG system, the In-set Model Ensemble Attack achieves better performance, but still significantly lower than the White-box Attack.

Interestingly, the Prompt-based Attack shows higher success than other black-box variants, yielding different success rates dependent on the generative model. While Gemini-2.5-Flash creates images that get retrieved more often, GPT-5’s images are better at generating the target answer. Overall, our results highlight that black-box attacks are not effective against VD-RAG systems in the targeted setting, among which, the Prompt-based Attack achieve the highest relative success.

The multiple target subvariant of the targeted attack optimizes the image to be retrieved and influence generation for a cluster of queries. When the image is retrieved, the VLM should generate the same answer for all of them. Therefore, the attack acts as an intermediate step between the base targeted attack and the universal attack. We target 5 queries: 1 attacker-chosen target query and its 4 nearest neighbors, computed by the Jina-Embeddings-V3 text embedding model. The rationale for using neighboring queries is to simulate the scenario in which the attacker wants to influence a range of queries related to a certain topic (e.g., elections, or a specific commercial product). The results confirm the findings from Setting I that attacks are more successful when CLIP-L is used, with success rates slightly lower than those in Setting I. Moreover, Prompt-based Attack are no longer useful when multiple queries are targeted.

In this setting, the attack targets multiple unrelated queries with the intent of generating a different malicious answer for each query using a single image. We target queries 1 and 2 in the dataset and use GPT-4o to generate corresponding malicious target answers. Table 3 shows slightly better results than Setting II but slightly worse than Setting I, where white-box attacks are successful against both legacy and SoTA embedding models in this challenging setting.

Focusing on the white-box attack, the universal attack produces images that are always retrieved for all queries (ASR-R@1 = 1) when the CLIP-L embedding model is used. To the contrary, state-of-the-art embedding models (ColPali-v1.3 and GME-Qwen2-VL-2B) never retrieve adversarial images as the top-1 relevant image but sometimes retrieve them within the top-5. Regarding generation, the universal attack consistently causes all VLMs to generate the target answer verbatim for almost all user queries in the test dataset. These results mirror those for the targeted attacks, where CLIP-L is the most vulnerable embedding model, while ColPali and GME remain robust to influences under all attacks.

The UMAP visualizations show a distinct modality gap in CLIP-L, however, a minimal gap for ColPali and GME. This illustrates the difficulty of generating a single image that is retrieved for all queries in these embedding spaces, leading to their observed robustness.

This distinction might explain why it is feasible to attack the CLIP-ViT-LARGE model. It is possible to create an artificial image that closely aligns with all queries, as its embeddings cluster in the same region.

We consider 4 losses: (1) MaxSim, which is the original metric used by Colpali, (2) AvgSim, which replaces the max operator by the average, (3) SoftMaxSim, which replaces the max operator by softmax, and (4) CosAvg, which computes the cosine similarity of the averaged token embeddings for both queries and images. Changing the similarity measure would degrade robustness, however, it is not wholly responsible for the robustness of ColPali.

For black-box attacks, those attacks are consistently unsuccessful against all model combinations. Even the In-set Model Ensemble Attack is only occasionally successful when CLIP-L is used.

The literature lacks specialized defenses against VD-RAG poisoning attacks. Furthermore, most of the defenses proposed for textual RAG are not straightforwardly applicable to multi-modal settings and incur a significant drop in benign performance. Nevertheless, we evaluate the resistance of the attacks to several defenses used by previous works. We only evaluate the white-box attacks since these are the most successful in both the targeted and universal settings.

This defense works by retrieving a larger number of KB items with the intention of diluting the effect of the retrieved adversarial image. We expand the number of retrieved images from 1 to 5 images when evaluating this defense.

The results show that expanding the retrieved knowledge (using k = 5) can degrade the attack performance if the attack was only trained using k = 1. However, an adaptive attack trained specifically against this value of k using 10% of the knowledge base, can effectively evade this defense when CLIP-L is the employed embedding model. This applies for both targeted and universal attacks. We conclude from these results that knowledge expansion on its own does not guarantee robustness of the RAG system against the attacks.

We use a VLM-as-a-Judge to evaluate the output on three metrics: (i) answer relevancy assesses if the answer is relevant to the query, (ii) context relevancy assesses if the retrieved images are relevant to the query, and (iii) answer faithfulness assesses if the answer is grounded in the retrieved images. We use the prompts proposed by Riedler & Langer (2024) to evaluate these metrics.

Qwen2.5-3B and InternVL3-2B as-a-judge demonstrate the capability to detect both universal and targeted attacks across all three metrics. SmolVLM-as-a-judge detects low answer relevancy for both attacks but performs worse in the other two metrics. These results show that adaptive attacks trained against the judge are able to bypass the defense, but there is no transferability of these attacks between judge models. We conclude that VLM-as-a-Judge is not able to improve the robustness of VD-RAG to the poisoning attacks.

As proposed by Shafran et al. (2024), we asked a state-of-the-art LLM, specifically Llama-3.1-8B-Instruct, to paraphrase all queries in the ViDoRe-V1-AI and ViDoRe-V2-ESG datasets and then use the paraphrased queries when evaluating whether the attacks are still successful.

Our results shows that query paraphrasing is not an effective defense against the attacks (computed against the original queries). Across both the targeted (Setting I) and universal attack objectives, both the ASR-R@1 and ASR-G_Sim@−1 remained the same as in the white-box attacks. With the only noteworthy exception being a reduction in targeted ASR-R@1 for ColPali, dropping from 0.60 to 0.20.

• Since we only inject one image into the KB, the attacks can be easily detected by majority vote-based methods. However, these methods often increase the latency of the system, and might degrade the benign performance.
• Extending the attacks to multiple images would likely improve the attack success rates, and is an interesting avenue for future work.
• Similarly, future work should investigate the robustness of these attacks against real-world perturbations (e.g., JPEG compression, watermarking) that might be applied to images before being added into the KB.
• Moreover, we could not evaluate the attacks against very large embedding models and VLMs due to compute constraints. Nevertheless, we conjecture that our attacks will succeed against larger models, as experimenting with model sizes between 256M to 4B parameters (spanning more than one order of magnitude) yielded very similar vulnerability.
• We have also conducted our experiments on open-weight models only, as white-box attacks could only be evaluated against those. Investigating the vulnerability of proprietary closed-weight models to black-box attacks is an interesting direction for future work.

In this paper we demonstrate the vulnerability of VD-RAG systems to poisoning attacks. The attacks show that conventional embedding models and VLMs are vulnerable to adversarial perturbation and that a single injected image is capable of either spreading disinformation on targeted topics, or causing a DoS against the entire RAG system (impacting both retrieval and generation). While both white-box and black-box attacks are successful in the targeted setting, only white-box attacks succeed in the universal setting. We also observe the notable adversarial robustness of the ColPali and GME embedding models in the universal attack case; however, they still prove vulnerable to more targeted attacks. Beyond vanilla VD-RAG pipelines, we evaluate several common RAG defenses and find them to be ineffective against the poisoning attacks considered. This work provides the first steps towards fully characterizing the vulnerabilities of visual document RAG systems and helps guide the development of more robust designs.