The Vistral-7B-Chat model is a powerful and state-of-the-art language model fine-tuned for function calling. In this blog post, we will delve into how to utilize this model effectively, allowing you to retrieve information such as the temperature of a specific city.

Understanding the Model

The Vistral-7B-Chat model was developed by hiieu and is based on the Viet-Mistral architecture. This model is a part of the Hugging Face family and is designed to interact with external functions seamlessly. Think of it as a highly intelligent assistant that can provide precise answers by tapping into predefined functions, much like a chef who knows exactly which spices to use for different dishes based on the flavors you desire.

Setup and Usage

To start using the Vistral-7B-Chat for function calling, you will need to follow these steps:

1. Install Required Libraries

Firstly, ensure that you have the Transformers library installed in your environment. You can do this using pip:

pip install transformers torch

2. Load the Model and Tokenizer

Next, you will load the model and tokenizer to prepare for interaction:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("hiieu/Vistral-7B-Chat-function-calling")
model = AutoModelForCausalLM.from_pretrained(
    "hiieu/Vistral-7B-Chat-function-calling",
    torch_dtype=torch.bfloat16,  # Change to torch.float16 if you're using V100
    device_map="auto",
    use_cache=True,
)

3. Define Functions and Conversation

Define the functions that the model can use and set up your conversation:

functions_metadata = [
    {
        "type": "function",
        "function": {
            "name": "get_temperature",
            "description": "Get temperature of a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description": "Name of the city"
                    }
                },
                "required": ["city"]
            }
        }
    }
]

conversation = [
    {"role": "system", "content": "Bạn là một trợ lý hữu ích có quyền truy cập vào các chức năng sau. Sử dụng chúng nếu cần."},
    {"role": "user", "content": "Thời tiết ở Hà Nội đang là bao nhiêu độ?"},
    {"role": "assistant", "content": "functioncall name: get_temperature, arguments: city: Hà Nội"}
]

4. Generate Responses

Finally, generate responses based on the conversation:

input_ids = tokenizer.apply_chat_template(conversation, return_tensors="pt").to(model.device)

out_ids = model.generate(
    input_ids=input_ids,
    max_new_tokens=768,
    do_sample=True,
    top_p=0.95,
    top_k=40,
    temperature=0.1,
    repetition_penalty=1.05
)

assistant = tokenizer.batch_decode(out_ids[:, input_ids.size(1):], skip_special_tokens=True)[0].strip()
print("Assistant:", assistant)  # This will print the assistant's response

Troubleshooting Tips

As you dive into using the Vistral-7B-Chat model, you may encounter some hurdles. Here are a few troubleshooting ideas:

• Error Loading Model: Ensure that the correct model name is used in the from_pretrained method.
• Out of Memory Issues: If you notice memory issues when generating outputs, try changing torch_dtype=torch.float16 to reduce memory usage.
• Incorrect Responses: If the model doesn’t provide correct outputs, verify your function definitions in functions_metadata.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.