ColPali: The Future of Visual Document Retrieval with PaliGemma-3B

Welcome to the exciting world of ColPali, a model designed to enhance document retrieval using advanced Vision Language Models (VLMs)! In this blog, we will walk you through how to use the ColPali model efficiently, troubleshoot common issues, and understand its underlying concepts without getting lost in technical jargon.

What is ColPali?

ColPali is a state-of-the-art model leveraging the architecture of PaliGemma-3B combined with a unique ColBERT strategy to generate multi-vector representations of texts and images. Think of it as a skilled librarian who not only remembers where every book is on the shelves but also can quickly pull out the pages you need based on visuals and context.

Getting Started with ColPali

Here’s how to effectively set up and use ColPali for your document retrieval needs:

1. Installation

• Ensure you have Python and pip installed.
• Install the ColPali engine by running:

  pip install colpali_engine==0.1.1

2. Running Inference

Using ColPali is straightforward once the engine is installed. Below is an example script to help you get started:

import torch
import typer
from torch.utils.data import DataLoader
from tqdm import tqdm
from transformers import AutoProcessor
from colpali_engine.models.paligemma_colbert_architecture import ColPali
from colpali_engine.trainer.retrieval_evaluator import CustomEvaluator
from colpali_engine.utils.colpali_processing_utils import process_images, process_queries
from colpali_engine.utils.image_from_page_utils import load_from_dataset

def main():
    # Load model
    model_name = "vidorecolpali"
    model = ColPali.from_pretrained("vidorecolpaligemma-3b-mix-448-base", torch_dtype=torch.bfloat16, device_map="cuda").eval()
    model.load_adapter(model_name)
    processor = AutoProcessor.from_pretrained(model_name)
    
    # Load images and queries
    images = load_from_dataset("vidoredocvqa_test_subsampled")
    queries = ["From which university does James V. Fiorca come?", "Who is the Japanese prime minister?"]

    # Run inference - docs
    dataloader = DataLoader(images, batch_size=4, shuffle=False, collate_fn=lambda x: process_images(processor, x))
    ds = []
    for batch_doc in tqdm(dataloader):
        with torch.no_grad():
            batch_doc = {k: v.to(model.device) for k, v in batch_doc.items()}
            embeddings_doc = model(**batch_doc)
        ds.extend(list(torch.unbind(embeddings_doc.to("cpu"))))

    # Run inference - queries
    dataloader = DataLoader(queries, batch_size=4, shuffle=False, collate_fn=lambda x: process_queries(processor, x, Image.new("RGB", (448, 448), (255, 255, 255))))
    qs = []
    for batch_query in dataloader:
        with torch.no_grad():
            batch_query = {k: v.to(model.device) for k, v in batch_query.items()}
            embeddings_query = model(**batch_query)
        qs.extend(list(torch.unbind(embeddings_query.to("cpu"))))

    # Run evaluation
    retriever_evaluator = CustomEvaluator(is_multi_vector=True)
    scores = retriever_evaluator.evaluate(qs, ds)
    print(scores.argmax(axis=1))

if __name__ == "__main__":
    typer.run(main)

In this script, we liken the ColPali model to an exceptional researcher who not only retrieves documents based on queries but also understands the context through visuals, making it far more efficient than traditional models. The model processes images and queries in batches for optimized performance, similar to how an efficient team works. By dividing tasks among members (or batches), they collectively reach the final result faster.

Troubleshooting Common Issues

While the setup and usage of ColPali are designed to be seamless, you might encounter some hiccups along the way. Here’s a helpful list of troubleshooting tips:

• Installation Errors: Ensure you have the correct version of Python and that pip is up to date. Use pip install --upgrade pip to update.
• Model Not Found: Double-check the model name and that the appropriate model files are downloaded correctly.
• Memory Issues: ColPali requires sufficient GPU memory. You may need to use a smaller batch size if you run into out-of-memory errors.
• Inference Failure: Ensure that your input images and queries are formatted correctly. Use the provided function to preprocess images and text before passing them to the model.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Limitations of ColPali

Although ColPali is a groundbreaking model, it has its limitations:

• The primary focus is on PDF-type documents and high-resource languages, which may restrict its application in other document types or less represented languages.
• There may be engineering efforts required to adapt the model’s multi-vector retrieval capabilities to other commonly used frameworks.

Final Thoughts

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.

By harnessing the power of ColPali, you can enhance your document retrieval tasks seamlessly. Embrace the future of AI-driven document management today!