Welcome to the exciting world of Ovis-1.6, our latest multi-modal large language model designed to enhance the way we process both text and visual data. In this guide, we’ll help you set up and run Ovis-1.6 effortlessly. Let’s dive in!

What is Ovis-1.6?

Ovis-1.6 is a sophisticated Multimodal Large Language Model (MLLM) that seamlessly aligns visual and textual embeddings. It is built upon Ovis-1.5 and boasts improved high-resolution image processing and advanced training techniques, making it a noteworthy advancement in AI technology.

Getting Started with Ovis-1.6

Installation

To start using Ovis-1.6, you need to install the necessary libraries. You can do this with a simple command:

bash
pip install torch==2.2.0 transformers==4.44.2 numpy==1.24.3 pillow==10.3.0

Loading the Model

Next, you will load the Ovis-1.6 model within your Python environment. Imagine you are preparing a dish, where loading the model is like gathering all your ingredients before cooking.

python
import torch
from PIL import Image
from transformers import AutoModelForCausalLM

# load model
model = AutoModelForCausalLM.from_pretrained(
    'AIDC-AIOvis1.6-Gemma2-9B',
    torch_dtype=torch.bfloat16,
    multimodal_max_length=8192,
    trust_remote_code=True).cuda()
text_tokenizer = model.get_text_tokenizer()
visual_tokenizer = model.get_visual_tokenizer()

Using Ovis-1.6 for Image-Text Interaction

Once you have loaded the model, you can start interacting with it using multimodal inputs. We visualize this process as creating a conversation between text and images. You will feed it an image and prompt, and the model will produce an output.

# enter image path and prompt
image_path = input("Enter image path: ")
image = Image.open(image_path)
text = input("Enter prompt: ")
query = f"image-to-text:{text}"

# format conversation
prompt, input_ids, pixel_values = model.preprocess_inputs(query, [image])

Batch Inference

If you want to analyze multiple images at once, Ovis-1.6 allows for batch processing, akin to preparing several dishes simultaneously. Just gather all your inputs and follow a similar process.

python
batch_inputs = [
    (example_image1.jpeg, "Describe the content of this image."),
    (example_image2.jpeg, "What is the equation in the image?")
]
batch_input_ids = []
batch_attention_mask = []
batch_pixel_values = []

for image_path, text in batch_inputs:
    image = Image.open(image_path)
    query = f"image-to-text:{text}"
    prompt, input_ids, pixel_values = model.preprocess_inputs(query, [image])
    
    # prepare for batch processing
    batch_input_ids.append(input_ids)
    batch_attention_mask.append(attention_mask)
    batch_pixel_values.append(pixel_values)

Troubleshooting

If you encounter any issues while using Ovis-1.6, consider the following troubleshooting steps:

• Ensure all dependencies are correctly installed as specified in the installation section.
• Check that your input image path is valid and accessible by the model.
• If you face memory issues, consider reducing the input resolution or batch size.

If problems persist, seek further assistance or explore our community resources. For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

Ovis-1.6 represents a great leap in the integration of visual and textual data processing. By following this guide, you should be well on your way to leveraging its capabilities. At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.

Additional Resources

For further reading and updates, check the official GitHub repository, access the live demo, and examine the research paper.