Are you excited about leveraging the InternVL2-40B-AWQ model for your image-text tasks? With its advanced quantization techniques and deployment capabilities, this powerful tool can significantly enhance your AI applications. In this blog, we will walk you through the setup, inference, and service deployment processes, ensuring a user-friendly experience.

Introduction to InternVL2-40B-AWQ

InternVL2-40B-AWQ utilizes the state-of-the-art AWQ algorithm for weight-only quantization and can drastically speed up inference time. Imagine you’re trying to make an enormous pizza (your model) as quickly as possible—cutting it into pieces (quantization) allows everyone to grab a slice much quicker!

Before we dive into the details, ensure you’ve installed the lmdeploy package:

pip install lmdeploy

Inference with InternVL2-40B-AWQ

Performing batched offline inference with the quantized model is straightforward. Below is the Python code you’ll need:

python
from lmdeploy import pipeline, TurbomindEngineConfig
from lmdeploy.vl import load_image

model = "OpenGVLab/InternVL2-40B-AWQ"
image = load_image("https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg")
backend_config = TurbomindEngineConfig(model_format='awq')

pipe = pipeline(model, backend_config=backend_config, log_level='INFO')
response = pipe(("describe this image", image))

print(response.text)

This code sets up a pipeline to use the InternVL2 model for inference. Think of this as placing your order at a restaurant—you provide your image (the dish) and receive a description (the meal) back!

Service Deployment

Once you’re satisfied with the inference results, it’s time to deploy the model as a service with just a command:

lmdeploy serve api_server OpenGVLab/InternVL2-40B-AWQ --backend turbomind --server-port 23333 --model-format awq

This simple command acts like the waiter who takes your hot, fresh pizza and delivers it to the table where you can serve it to your guests (your users) quickly and efficiently!

API Access: Interacting with Your Service

If you want to interact with your deployed model using an OpenAI-style interface, first, install the OpenAI package:

pip install openai

Then, use the following code snippet to make API calls:

from openai import OpenAI

client = OpenAI(api_key='YOUR_API_KEY', base_url='http://0.0.0.0:23333/v1')
model_name = client.models.list().data[0].id

response = client.chat.completions.create(
    model=model_name,
    messages=[
        {
            "role": "user",
            "content": {
                "type": "text",
                "text": "describe this image"
            },
            "type": "image_url",
            "image_url": {
                "url": "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/tiger.jpeg"
            }
        }
    ],
    temperature=0.8,
    top_p=0.8
)

print(response)

This segment allows you to communicate with your model as if you’re sending a text message to a friend—quick and convenient!

Troubleshooting Tips

If you encounter any issues during installation or inference, consider the following tips:

• Ensure your GPU is compatible with the model requirements. Supported NVIDIA GPUs include Turing, Ampere, and Ada Lovelace architectures.
• Check that all packages are installed correctly and are updated to the latest versions.
• Verify that your image URL is accessible and correctly formatted.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.