If you’ve ever wanted to dive into the exciting world of advanced language models, you’re in the right place! Qwen2 offers a series of large language models, and today, we’ll be focusing on how to use the instruction-tuned 1.5B Qwen2 model. In this guide, we’ll break down how to get it up and running smoothly, even if you’re a newcomer.

Understanding Qwen2

Picture Qwen2 as a multi-layered cake of programming prowess—each layer representing different functionalities and capabilities, from language understanding to coding and reasoning. This complex cake can be used in numerous applications, and the 1.5B version is like the sweet spot, balancing power and efficiency.

Requirements

• It’s advisable to clone the llama.cpp repository and follow the official installation guide.
• Make sure you have Python and pip installed for package management.

How to Use Qwen2

1. Cloning or Downloading the Model

To get the model, you can either clone the GitHub repository or download the required GGUF file directly by using the following command:

huggingface-cli download Qwen/Qwen2-1.5B-Instruct-GGUF qwen2-1_5b-instruct-q5_k_m.gguf --local-dir . --local-dir-use-symlinks False

2. Running Qwen2

To run the Qwen2 model, we recommend using the `llama-server`, as it offers simplicity and compatibility with the OpenAI API. Use the following command:

./llama-server -m qwen2-1_5b-instruct-q5_k_m.gguf -ngl 28 -fa

Here, -ngl 28 refers to offloading 28 layers to GPUs and -fa stands for using flash attention.

3. Accessing the Deployed Service

You can then access the deployed service with OpenAI API using the following Python code:

import openai
client = openai.OpenAI(
    base_url="http://localhost:8080/v1", # "http://:port"
    api_key = "sk-no-key-required"
)
completion = client.chat.completions.create(
    model="qwen",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "tell me something about michael jordan"}
    ]
)
print(completion.choices[0].message.content)

Troubleshooting Tips

• If you encounter issues while running the server, make sure your ports are correctly set and not blocked by a firewall.
• Check for any dependency errors, especially with installed libraries in Python.
• If you’re struggling to understand how to relay messages correctly, double-check the format of your input messages as shown above.
• For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Final Thoughts

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.