In this blog, we’ll walk you through the steps of utilizing the InternVL2-Llama3-76B-AWQ model for image-text processing. This process includes setting up the environment, implementing the model for inference, and deploying it as a service. Whether you’re an AI enthusiast or a seasoned developer, you’ll find this guide user-friendly!

Table of Contents

• Quick Start
• Inference
• Service

Quick Start

Before diving into inference, you need to ensure you have lmdeploy installed. You can install it using pip:

pip install lmdeploy

Inference

Now that you have everything set up, you can conduct batched offline inference using the quantized model. It’s like having a smart friend who quickly analyzes images and responds based on your questions!

Here’s how to do it:

python
from lmdeploy import pipeline, TurbomindEngineConfig
from lmdeploy.vl import load_image

model = "OpenGVLabInternVL2-Llama3-76B-AWQ"
image = load_image("https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg")
backend_config = TurbomindEngineConfig(model_format='awq')
pipe = pipeline(model, backend_config=backend_config, log_level='INFO')
response = pipe(("describe this image", image))
print(response.text)

In the analogy, think of the entire process like preparing a meal. The model is your chef, the image is the raw ingredient, and you’re providing instructions to create a delicious output (the description).

Service

Once you have the inference model ready, the next step is to deploy it as a service. Imagine you’ve trained your chef so well that now you can call for their services anytime!

Start the API server by running the command below:

lmdeploy serve api_server OpenGVLabInternVL2-Llama3-76B-AWQ --backend turbomind --server-port 23333 --model-format awq

To make calls using the OpenAI-style interface, ensure you have the openai package installed:

pip install openai

Then, implement the following code to make an API call:

python
from openai import OpenAI

client = OpenAI(api_key='YOUR_API_KEY', base_url='http://0.0.0.0:23333/v1')
model_name = client.models.list().data[0].id
response = client.chat.completions.create(
    model=model_name,
    messages=[
        {
            "role": "user",
            "content": {
                "type": "text",
                "text": "describe this image",
            },
            "type": "image_url",
            "image_url": {
                "url": "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/tiger.jpeg",
            },
        },
    ],
    temperature=0.8,
    top_p=0.8
)
print(response)

Troubleshooting

• If you encounter issues during the installation of lmdeploy, ensure your Python version is compatible and has the necessary permissions to install packages.
• For errors in the API server startup, check that the specified port (23333) is not occupied by any other service.
• Having trouble with image URLs? Ensure that the provided URL is accessible and valid.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

By following the steps outlined above, you can successfully utilize the InternVL2-Llama3-76B-AWQ model for image-text processing. Engaging with AI models can feel complicated, but once you break it down into manageable tasks, it becomes effortless!

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.