The Tess-3-7B model is a specialized variant of the Mistral-7B-v0.3 base model, designed for natural language processing tasks. In this guide, we will walk you through the steps to set up and run inference with the Tess-3-7B model, tackling any potential issues you may encounter along the way.

What You Need to Get Started

• Python installed on your machine
• The Tess-3-7B model downloaded and accessible
• Pytorch and the transformers library

Setting Up Your Environment

Before diving into the code, ensure that your environment is properly set up. You can run the following command to install the necessary libraries:

pip install torch transformers stop-word

Running Inference with the Tess-3-7B Model

Now it’s time to run inference. Here’s a simplified analogy to explain how the code works: imagine you are a chef trying to prepare a special dish using a unique recipe. The ingredients represent the variables in your code, and following the steps is like executing the code itself. The output of your dish is similar to the generated text produced by the model.

Step-by-Step Code Explanation

import torch, json
from transformers import AutoModelForCausalLM, AutoTokenizer
from stop_word import StopWordCriteria

model_path = "migtissera/Tess-3-7B-SFT"
output_file_path = "/home/migel/conversations.jsonl"

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.float16,
    device_map="auto",
    load_in_4bit=False,
    trust_remote_code=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
terminators = [tokenizer.convert_tokens_to_ids("