Multi-Modal RAG

TL;DR

Real documents aren't pure text — PDFs have charts, diagrams, tables, screenshots.
The simplest, most reliable approach: turn images into text with a vision model, then embed everything the normal way.
A query like "what drove Q3 growth?" can then retrieve a chart's caption, not just paragraphs.

A normal RAG pipeline only reads text, so it's blind to the chart that actually answers the question. Multi-modal RAG makes images retrievable too. The most robust pattern is to describe each image as text and put it in the same vector store as the paragraphs. One submodule per step, ending with a cheat sheet.

Multimodal RAG: extract text, caption images with a vision model, embed both into one store

Reading the diagram: a PDF page has both paragraphs and a chart image. The text is extracted directly; the chart is sent to a vision model that writes a caption. Both become text, embed into one shared space, and at query time the right one is retrieved.

Why plain RAG misses images

Take a one-page report: text describing revenue, plus a bar chart showing Q1→Q3 growth. Standard extraction (PyPDFLoader) pulls the words but drops the chart — so a question like "which quarter grew fastest?" finds nothing, even though the chart says it plainly. The fix is to give the chart a text representation the retriever can match against.

Describe images with a vision model

Send each extracted image to a vision-capable model and ask for a factual caption. That caption is the image's searchable text.

import base64
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage

vision = ChatOpenAI(model="gpt-4o")   # a vision-capable model

def describe_image(image_bytes):
    b64 = base64.b64encode(image_bytes).decode()
    msg = HumanMessage(content=[
        {"type": "text", "text":
            "Describe this image factually for search: any chart type, "
            "axes, trends, and key numbers."},
        {"type": "image_url",
         "image_url": {"url": f"data:image/png;base64,{b64}"}},
    ])
    return vision.invoke([msg]).content
# → "Bar chart of revenue by quarter: Q1 moderate, Q2 higher,
#    Q3 highest with exponential growth."

The model converts a picture into the kind of text your embedding model already understands.

Extract text and images from the PDF

Pull both streams out of the document, then run images through the captioner:

import fitz  # PyMuPDF

doc = fitz.open("annual_revenue.pdf")
text_docs, image_captions = [], []

for page in doc:
    text_docs.append(page.get_text())                 # the paragraphs
    for img in page.get_images(full=True):
        xref = img[0]
        image_bytes = doc.extract_image(xref)["image"]
        image_captions.append(describe_image(image_bytes))  # caption each image

Embed both into one store

Captions are just text, so they go into the same vector store as the paragraphs — no special multimodal index needed.

from langchain_community.vectorstores import FAISS
from langchain_huggingface import HuggingFaceEmbeddings

emb = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
all_text = text_docs + image_captions          # paragraphs + image captions together
vectorstore = FAISS.from_texts(all_text, emb)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

Answer from whichever is relevant

At query time, retrieval just works — the question matches the chart's caption as easily as a paragraph, and the LLM answers from both.

docs = retriever.invoke("What drove the strongest revenue growth?")
# retrieves the chart caption ("Q3 highest, exponential growth") +
# the surrounding text ("Q3 had exponential growth due to global expansion")
# → LLM answers: "Q3, driven by global expansion."

Cheat sheet

Step	Code
Open PDF + iterate pages	`fitz.open(path)` → `for page in doc`
Extract text	`page.get_text()`
Extract images	`page.get_images(full=True)` → `doc.extract_image(xref)`
Caption an image	vision model + base64 `image_url` message
Embed text + captions together	`FAISS.from_texts(text_docs + image_captions, emb)`

⚠ Common mistakes

Using a text-only loader and silently dropping every chart/diagram — the most common multimodal bug.
Vague caption prompts — ask for chart type, axes, trends, and numbers, or the caption won't match real questions.
Forgetting captions cost vision-model calls at index time — fine, but budget for it on large image-heavy corpora.
Assuming you need a special "multimodal embedding" — describing images as text and reusing your normal text embeddings is simpler and works well.

Quick self-check

Why does standard RAG fail on a PDF with charts?

Text-only extraction drops the images, so anything the chart conveys is invisible to retrieval — the answer simply isn't in the index.

What's the core trick of this multimodal approach?

Convert each image into a text caption with a vision model, then embed it like any other text — so one normal vector store covers both modalities.

Do you need a special multimodal vector index?

No. Once images are captioned as text, your normal text embeddings and store handle everything.

When are image captions generated — index time or query time?

Index time (during ingestion). Queries then just retrieve the stored captions like any other text.

Next: AI Agents & Agentic AI — coming soon (studying next).

Why plain RAG misses images​

Describe images with a vision model​

Extract text and images from the PDF​

Embed both into one store​

Answer from whichever is relevant​

Cheat sheet​