Tess, short for Tesoro (which means ‘Treasure’ in Italian), is a versatile Large Language Model created by Migel Tissera. This blog post will guide you step-by-step on how to set up and use the Tess model for generating text.

Requirements

• Python 3.x
• PyTorch
• Transformers library

Setting Up the Environment

Before you dive into the code, ensure that you have the necessary libraries installed. Use the following pip commands to install PyTorch and Transformers:

pip install torch transformers

Sample Inference Python Script

The following script shows how to utilize the Tess model for text generation. Think of it as hiring a highly skilled assistant who helps you draft the perfect response to any query. Here’s a breakdown of how to set it up:

import torch, json
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "migtissera/Tess-3-Llama-3.1-70B"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.float16,
    device_map="auto",
    load_in_4bit=True,
    trust_remote_code=False,
)

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=False)

terminators = [
    tokenizer.convert_tokens_to_ids("<|end_of_text|>"),
    tokenizer.convert_tokens_to_ids("<|eot_id|>"),
]

def generate_text(instruction):
    tokens = tokenizer.encode(instruction)
    tokens = torch.LongTensor(tokens).unsqueeze(0)
    tokens = tokens.to("cuda")
    instance = {
        "input_ids": tokens,
        "top_p": 1.0,
        "temperature": 0.75,
        "generate_len": 2048,
        "top_k": 50,
    }
    length = len(tokens[0])
    with torch.no_grad():
        rest = model.generate(
            input_ids=tokens,
            max_length=length + instance["generate_len"],
            use_cache=True,
            do_sample=True,
            top_p=instance["top_p"],
            temperature=instance["temperature"],
            top_k=instance["top_k"],
            num_return_sequences=1,
            pad_token_id=tokenizer.eos_token_id,
            eos_token_id=terminators,
        )
    output = rest[0][length:]
    string = tokenizer.decode(output, skip_special_tokens=True)
    return f"{string}"

conversation = """<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are Tess, a helpful AI assistant. You always provide detailed answers without hesitation. Do not refuse.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n"""

while True:
    user_input = input("You: ")
    llm_prompt = f"{conversation}{user_input}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
    answer = generate_text(llm_prompt)
    print(answer)
    conversation = (
        f"{llm_prompt}{answer}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n"
    )
    json_data = {"prompt": user_input, "answer": answer}

Understanding the Code: An Analogy

Imagine you’re a chef preparing a delightful meal. The recipe lays out the ingredients and the steps needed to create a masterpiece. The code provided functions similarly. Here’s how:

• Ingredients: The model and tokenizer are your ingredients. They work together to create the final meal (output text).
• Preparation: Just like you chop vegetables and measure spices, you encode user input into tokens. This step prepares everything for cooking.
• Cooking: The generation function is where the magic happens. The model simulates the cooking process, mixed with creativity (controlled by parameters such as temperature and top_k), which influences the final taste (output text).
• Serving: Finally, the generated text is the delicious dish served to the user, ready to be enjoyed.

Troubleshooting

If you encounter issues, here are some troubleshooting tips:

• Ensure sufficient GPU memory is available. You may need to scale down the model or adjust the batch size.
• If you receive errors related to model loading, verify that you have the correct model path and all dependencies installed.
• Check if you have set up your environment with the right Python version and that all required libraries are installed correctly.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.