LAVIS is an enticing Python deep learning library designed for LAnguage-and-VISion intelligence research and applications. It aims to serve as a one-stop powerhouse for engineers and researchers to rapidly develop models for specific multimodal scenarios and benchmark them with ease.

What’s New

With every update, LAVIS rolls out new models to enhance the user experience. Let’s take a look at some exciting releases:

• November 2023: Implementation of X-InstructBLIP – A cross-modality framework that integrates various modalities without extensive customization. [Paper](https://arxiv.org/pdf/2311.18799.pdf) | [Project Page](https://github.com/salesforce/LAVIS/tree/main/projects/xinstructblip)
• July 2023: Introduction of BLIP-Diffusion – A text-to-image generation model designed to train significantly faster than its predecessor. [Paper](https://arxiv.org/abs/2305.06500) | [Project Page](https://github.com/salesforce/LAVIS/tree/main/projects/blip-diffusion)
• May 2023: Release of InstructBLIP – A novel framework utilizing BLIP-2 models for accomplishing a range of vision-language tasks. [Paper](https://arxiv.org/abs/2305.06500) | [Project Page](https://github.com/salesforce/LAVIS/tree/main/projects/instructblip)
• January 2023: BLIP-2 delivers a generic strategy for vision-language pretraining. [Paper](https://arxiv.org/abs/2301.12597) | [Project Page](https://github.com/salesforce/LAVIS/tree/main/projects/blip2)

Installation Guide

Ready to get started? Follow these steps:

1. Creating a Conda Environment (Optional):
   Run the following commands:
   

   conda create -n lavis python=3.8
   conda activate lavis

2. Install from PyPI:
   Execute:
   

   pip install salesforce-lavis

3. Or build from source:
   If you’re looking to contribute, you can clone the repository:
   

   git clone https://github.com/salesforce/LAVIS.git
   cd LAVIS
   pip install -e .

Getting Started

Once you have LAVIS installed, it’s time to explore its potential:

Model Zoo

To see the supported models in LAVIS:

from lavis.models import model_zoo
print(model_zoo)

Image Captioning Example

Imagine your model as a talented photographer, keen to take in the beauty of an image and describe it eloquently. Here’s how you’d generate a caption for an image:

import torch
from lavis.models import load_model_and_preprocess

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model, vis_processors, _ = load_model_and_preprocess(name="blip_caption", model_type="base_coco", is_eval=True, device=device)
raw_image = Image.open("docs/static/merlion.png").convert("RGB")
image = vis_processors["eval"](raw_image).unsqueeze(0).to(device)
caption = model.generate(image)
print(caption)  # Outputs: [a large fountain spewing water into the air]

Think of an image-capable entity like a concierge at a luxury hotel. It not only identifies the beautiful sights but can eloquently summarize them in a few simple sentences. This helps you gain a clearer understanding without needing to witness everything firsthand.

Visual Question Answering (VQA)

Want to ask questions about the images? Here’s how you can enable AI to act like a clever tour guide:

question = "Which city is this photo taken?"
model, vis_processors, txt_processors = load_model_and_preprocess(name="blip_vqa", model_type="vqav2", is_eval=True, device=device)
image = vis_processors["eval"](raw_image).unsqueeze(0).to(device)
question = txt_processors["eval"](question)
answer = model.predict_answers(samples=image, text_input=question, inference_method="generate")
print(answer)  # Outputs: [singapore]

Troubleshooting

If you encounter any issues during installation or usage, here are a few helpful troubleshooting tips:

• Ensure that you have the latest version of Python and pip installed.
• Verify that your dependencies (like PyTorch) are properly installed and compatible.
• If your models are not loading correctly, double-check the names used when calling the model loading functions.
• For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Always remember to review the logs for errors, as they can provide useful hints about what might be going wrong!

Conclusion

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.