The TrustLLM toolkit offers a comprehensive framework aimed at assessing the trustworthiness of large language models (LLMs). In this guide, you will learn how to install, download datasets, generate data, and evaluate the performance of your models using TrustLLM. Let’s embark on this journey toward understanding the intricate world of trust in AI.

About TrustLLM

TrustLLM is designed to study various dimensions of trustworthiness in LLMs, outlining several key principles and benchmarks. By employing this toolkit, you will gain insights into the performance of mainstream LLMs across multiple datasets, focusing on aspects such as truthfulness, safety, fairness, and more. For more information, visit the TrustLLM Project Website.

Before Evaluation

Installation

To get started with TrustLLM, you have several installation options:

• Via GitHub (recommended): git clone git@github.com:HowieHwong/TrustLLM.git
• Via pip: pip install trustllm
• Via conda:
  • Create a new environment: conda create --name trustllm python=3.9
  • Install required packages: cd trustllm_pkg && pip install .

Dataset Download

After installation, you will want to download the TrustLLM dataset:

python
from trustllm.dataset_download import download_dataset
download_dataset(save_path=save_path)

Generation

TrustLLM provides an easy way for data generation. From version 0.2.0, you can utilize the generation section by referring to this page: Generation Details. Here is an example code snippet:

python
from trustllm.generation.generation import LLMGeneration
llm_gen = LLMGeneration(
    model_path='your model name',
    test_type='test section',
    data_path='your dataset file path',
    model_name='model_name',
    online_model=False,
    use_deepinfra=False,
    use_replicate=False,
    repetition_penalty=1.0,
    num_gpus=1,
    max_new_tokens=512,
    debug=False,
    device='cuda:0'
)
llm_gen.generation_results()

Evaluation

Evaluating the trustworthiness of your LLM can be conducted using the provided toolkit. It offers vast functionalities to assess models. Here’s a simple example to run a truthfulness evaluation:

python
from trustllm.task.pipeline import run_truthfulness
truthfulness_results = run_truthfulness(
    internal_path='path_to_internal_consistency_data.json',
    external_path='path_to_external_consistency_data.json',
    hallucination_path='path_to_hallucination_data.json',
    sycophancy_path='path_to_sycophancy_data.json',
    advfact_path='path_to_advfact_data.json'
)

Troubleshooting

While navigating through the installation and evaluation process, you might encounter some common issues:

• Installation Errors: Ensure you are using the correct version of Python and have installed all package dependencies.
• Dataset Download Issues: Verify the save path is accessible and correct.
• Evaluation Output Problems: Double-check the paths you are providing in the evaluation functions.

If you are still facing issues, for more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.