If you’re a coding enthusiast or working with language models, you might have heard of the latest release: Qwen1.5-7B-Chat-GPTQ-Int8. In this guide, we will walk through the quickstart process of using this powerful language model, ensuring you are equipped to harness its capabilities seamlessly.

What is Qwen1.5?

Qwen1.5 is a beta version of the Qwen2 language model, known for its transformer-based, decoder-only architecture. This model boasts several enhancements over its predecessor, including:

• Multiple sizes to choose from: 0.5B, 1.8B, 4B, 7B, 14B, and 72B.
• Improved human preference for chat models.
• Support for multiple languages.
• Capable of handling a 32K context length across all model sizes.
• No need for trust_remote_code.

For more details, you can read our blog post and visit the GitHub repo.

Model Details

In this series, Qwen1.5 includes different sized decoder language models, each with a base language model and a chat model. It employs advanced features like:

• SwiGLU activation.
• Attention QKV bias.
• Group query attention.
• Adaptive tokenizer for multiple natural languages and codes.

However, keep in mind that the beta version doesn’t yet include GQA and the mixture of sliding window and full attention functionalities.

Getting Set Up

To get started using Qwen1.5, you need to ensure you have the right environment. We recommend using the latest version of Hugging Face’s transformers (version 4.37.0). Failing to do so may present you with a frustrating KeyError: qwen2.

Quickstart Guide

Follow these steps to quickly load the model and generate content:

python
from transformers import AutoModelForCausalLM, AutoTokenizer

device = 'cuda'  # the device to load the model onto
model = AutoModelForCausalLM.from_pretrained(
    'Qwen/Qwen1.5-7B-Chat-GPTQ-Int8',
    torch_dtype='auto',
    device_map='auto'
)
tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen1.5-7B-Chat-GPTQ-Int8')

prompt = 'Give me a short introduction to large language model.'
messages = [
    {'role': 'system', 'content': 'You are a helpful assistant.'},
    {'role': 'user', 'content': prompt}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

model_inputs = tokenizer([text], return_tensors='pt').to(device)
generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

Imagine you are a chef preparing a meal for the first time. You carefully gather your ingredients (loading the model), set the kitchen temperature (setting the device), and then follow a recipe step-by-step to create a delicious dish (generating text). Just as in cooking, following each step accurately is crucial to achieving the perfect outcome with your model!

Troubleshooting Tips

Occasionally, you might run into issues while working with Qwen1.5. Here are some common troubleshooting suggestions:

• If you experience instances of code switching or erratic behavior in responses, consider utilizing the hyper-parameters provided in generation_config.json.
• Make sure your environment is equipped with all required dependencies, especially the correct version of transformers.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

With these guidelines, you’re ready to dive into the world of Qwen1.5. This framework is packed with potential to assist you in various projects involving text generation and natural language processing.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.