Ovis-1.6 A New Milestone in Multimodal Language Models

In the rapidly advancing field of artificial intelligence, the unveiling of Ovis-1.6 is a noteworthy development. This open-source, multimodal large language model (MLLM) is engineered to fluidly integrate both visual and textual data, providing a powerful tool for developers and researchers. Let's explore what sets Ovis-1.6 apart in the landscape of AI models.

Pioneering Multimodal Fusion

Ovis-1.6 builds on the foundation laid by Ovis1.5, enhancing its capabilities to process high-resolution images while being trained on a more diverse and refined dataset. Its innovative architecture harmonizes visual and textual embeddings, allowing it to generate content that is coherent across both modalities.

Notable Advancements in Ovis-1.6

- Enhanced Image Processing: With its cutting-edge high-resolution image processing capabilities, Ovis-1.6 excels at handling intricate visual data.

- Expanded Training Dataset: By training on a larger and more varied dataset, the model improves its ability to generalize across multiple domains.

- Optimized Training Methodology: Utilizing Differentially Private Optimization (DPO) post-instruction tuning, Ovis-1.6 ensures robust performance with a focus on privacy.

Unparalleled Performance

Despite its relatively modest size of 10 billion parameters, Ovis-1.6-Gemma2-9B surpasses many open-source MLLMs with as many as 30 billion parameters, leading the OpenCompass benchmark. This achievement highlights the model's efficiency and capability.

Practical Applications

Ovis-1.6 is designed for real-world utility, not just theoretical potential. From generating detailed descriptions based on images to interpreting complex visual data, this model is equipped to tackle a wide array of tasks with remarkable precision.

Getting Started with Ovis-1.6

For those eager to delve into Ovis-1.6, the model is available on platforms such as [GitHub](https://github.com/AIDC-AIOvis) and [Hugging Face](https://huggingface.co/spaces/AIDC-AIOvis/1.6-Gemma2-9B). Here’s a brief guide to get you started:

Installation and Setup

To begin working with Ovis-1.6, install the necessary packages:

Copy
pip install torch==2.2.0 transformers==4.44.2 numpy==1.24.3 pillow==10.3.0

Running the Model

Below is a code snippet to demonstrate how to run Ovis with multimodal inputs:

Copy
import torch
from PIL import Image
from transformers import AutoModelForCausalLM

# Load model
model = AutoModelForCausalLM.from_pretrained("AIDC-AIOvis/1.6-Gemma2-9B",
                                             torch_dtype=torch.bfloat16,
                                             multimodal_max_length=8192,
                                             trust_remote_code=True).cuda()
text_tokenizer = model.get_text_tokenizer()
visual_tokenizer = model.get_visual_tokenizer()

# Enter image path and prompt
image_path = input("Enter image path: ")
image = Image.open(image_path)
text = input("Enter prompt: ")
query = f"{image} {text}"

# Format conversation
prompt, input_ids, pixel_values = model.preprocess_inputs(query, [image])
attention_mask = torch.ne(input_ids, text_tokenizer.pad_token_id)
input_ids = input_ids.unsqueeze(0).to(device=model.device)
attention_mask = attention_mask.unsqueeze(0).to(device=model.device)
pixel_values = [pixel_values.to(dtype=visual_tokenizer.dtype, device=visual_tokenizer.device)]

# Generate output
with torch.inference_mode():
    gen_kwargs = dict(
        max_new_tokens=1024,
        do_sample=False,
        top_p=None,
        top_k=None,
        temperature=None,
        repetition_penalty=None,
        eos_token_id=model.generation_config.eos_token_id,
        pad_token_id=text_tokenizer.pad_token_id,
        use_cache=True
    )
    output_ids = model.generate(input_ids, pixel_values=pixel_values, attention_mask=attention_mask, **gen_kwargs)[0]
    output = text_tokenizer.decode(output_ids, skip_special_tokens=True)
    print(f"Output:\n{output}")

For comprehensive usage instructions, including an inference wrapper and Gradio UI, visit the [Ovis GitHub repository](https://github.com/AIDC-AIOvis?tab=readme-ov-file#inference).

Licensing and Acknowledgments

Ovis-1.6 is distributed under the Apache License, Version 2.0, supporting both personal and commercial use. The development team has integrated compliance-checking algorithms to adhere to copyright standards, and users are encouraged to report any issues.

Conclusion

Ovis-1.6 signifies a major advancement in the fusion of visual and textual data, offering a versatile platform for diverse applications. Whether you are a researcher, developer, or AI enthusiast, this model provides a powerful avenue for exploring the future of multimodal AI.