How to Create and Use SaigaLlama3: A Russian Chatbot

July 8, 2024

In this blog post, we’ll walk you through setting up and using SaigaLlama3, an 8B Russian Llama-3-based chatbot. This chatbot relies on the Llama-3 model and is adaptable to various use cases, making it ideal for language processing tasks.

Getting Started with SaigaLlama3

To begin, make sure you have access to the required dependencies, including PyTorch and Hugging Face’s `transformers` library. Let’s get started with the installation and configuration of the model.

Installation

Before running your code, ensure your environment is ready. Here is an outline of how to set up the chatbot model:

Ensure you have Python and pip installed.
Install PyTorch and the transformers library by running:

pip install torch transformers

Using SaigaLlama3 Model

The following Python snippet demonstrates how to load the model and generate responses. Think of the code as a recipe for a cake. Each line of code represents an ingredient essential for the final product.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig

MODEL_NAME = "IlyaGusev/saiga_llama3_8b"
DEFAULT_SYSTEM_PROMPT = "Ты — Сайга, русскоязычный автоматический ассистент..."

model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, load_in_8bit=True, torch_dtype=torch.bfloat16, device_map='auto')
model.eval()

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
generation_config = GenerationConfig.from_pretrained(MODEL_NAME)

inputs = ["Почему трава зеленая?", "Сочини длинный рассказ, обязательно упоминая следующие объекты. Дано: Таня, мяч"]
for query in inputs:
    prompt = tokenizer.apply_chat_template([
        {"role": "system", "content": DEFAULT_SYSTEM_PROMPT},
        {"role": "user", "content": query}
    ], tokenize=False, add_generation_prompt=True)
    data = tokenizer(prompt, return_tensors='pt', add_special_tokens=False)
    data = {k: v.to(model.device) for k, v in data.items()}
    output_ids = model.generate(**data, generation_config=generation_config)[0]
    output_ids = output_ids[len(data['input_ids'][0]):]
    output = tokenizer.decode(output_ids, skip_special_tokens=True).strip()
    print(query)
    print(output)
    print("------------------------------")

In this recipe:

Importing Libraries: Just like gathering ingredients, we import necessary modules.
Model Import: The model is loaded using its name, like taking eggs from the fridge.
Generating Prompt: We prepare the input prompt, similar to mixing your cake batter.
Data Processing: Data transformations allow the model to digest the inputs.
Model Generation: Finally, we bake our cake (generate the output) and print the delicious results.

Prompt Formats

Two different prompt formats are recognized by the SaigaLlama3 model:

Original Llama-3 Format:

begin_of_text
        start_header_id
        system
        end_header_id
        Ты — Сайга, русскоязычный автоматический ассистент.
        end_header_id
        start_header_id
        user
        end_header_id
        Как дела?
        end_header_id
        start_header_id
        assistant
        end_header_id
        Отлично, а у тебя?
        end_header_id
        ...

ChatML Format:

im_startsystem
        Ты — Сайга, русскоязычный автоматический ассистент.
        im_end
        im_startuser
        Как дела?
        im_end
        im_startassistant
        Отлично, а у тебя?
        im_end
        ...

Troubleshooting Tips

If you encounter issues while using SaigaLlama3, here are some suggestions:

Ensure you have the correct versions of dependencies installed.
Check that the model path and prompt formats align with the examples provided.
Review any error messages for guidance on what might be wrong.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

This blog provided an overview of the SaigaLlama3 chatbot, including installation, usage, and troubleshooting tips. At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.