HumanPrompt is an innovative framework designed for simplifying the human-in-the-loop design, management, sharing, and usage of prompts and their methods. This blog post will guide you through the essential steps to get started with HumanPrompt, how to accelerate your research, and what to do if you run into trouble.

To Start

Getting started with HumanPrompt is simple. Just follow these steps:

• Clone the repository and install the HumanPrompt package by running:

bash
pip install -e .

• This sets up everything you’ll need and creates a soft link hub to your .humanprompt/artifacts/hub.
• Next, set your OpenAI API key by running:

bash
export OPENAI_API_KEY=YOUR_OPENAI_API_KEY

• You can then start experimenting with the repository.

Using the Framework

HumanPrompt allows you to experiment flexibly with various methods. Think of it as a toolkit for assembling a custom bicycle. You choose the frame (your method), add wheels (data), and modify your ride as per your needs.

For example, to run a Chain-of-Thought method on CommonsenseQA, you could follow this script:

python
from humanprompt.methods.auto.method_auto import AutoMethod
from humanprompt.tasks.dataset_loader import DatasetLoader

# Get one built-in method
method = AutoMethod.from_config(method_name='cot')

# Load dataset and select example
data = DatasetLoader.load_dataset(dataset_name='commonsense_qa', dataset_split='test')
data_item = data[0]

# Adapt data for input
data_item['context'] = "Answer choices: " + ". ".join(
    ["{}: {}".format(label.lower(), text.lower()) for label, text in zip(data_item['choices']['label'], data_item['choices']['text'])]
)

# Run the method
result = method.run(data_item)
print(result)
print(data_item)

Consider that you’re focusing a camera lens on a specific scene. Each command in the script is a twist of the lens, adjusting the focus to capture a clear image of what you want to analyze.

To Accelerate Your Research

Here are some tips to help you speed up your research with the HumanPrompt framework:

Config

HumanPrompt operates under a one-config, one-experiment paradigm, making it easy to test various prompting methods. You can create a configuration file in YAML format to set your parameters. Here’s a template:

yaml
---
dataset:
  dataset_name: gsm8k
  dataset_split: test
  dataset_key_map:
    question: question
    answer: answer

method:
  method_name: cot
  method_args:
    client_name: openai
    transform: cot.gsm8k.transform_cot_gsm8k.CoTGSM8KTransform
    extract: cot.gsm8k.extract_cot_gsm8k.CoTGSM8KExtract
    extraction_regex: .*The answer is (.*).n?
    prompt_file_path: cot_gsm8k_prompt.txt
    max_tokens: 512
    temperature: 0
    engine: code-davinci-002
    stop_sequence: nn

metrics:
  - exact_match

Run Experiments

Once you have your configuration file, you can run your experiment using the following command:

bash
python run_experiment.py --exp_name cot-gsm8k --num_test_samples 300

Architecture Overview

The structure of the HumanPrompt repository is organized to facilitate easy navigation and usage. Here’s an overview:

• configs: Contains configuration files for experiments.
• methods: Holds various prompting methods, each typically related to a research paper.
• tasks: Includes dataset loading and preprocessing scripts.
• artifacts: Contains the building blocks for the complete prompting pipeline.

Contributing

The HumanPrompt framework is built for community contributions. You can add your methods by following these steps:

1. Clone the repository.
2. Create a new branch from the main branch.
3. Add your method and necessary files.
4. Run a demo using your method.
5. Submit a Pull Request.

Troubleshooting

If you encounter issues, consider these troubleshooting steps:

• Ensure all dependencies are installed correctly.
• Recheck your environmental variables, especially the OpenAI API key.
• If running experiments fails, confirm that your config files are correctly formatted.
• For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.

Conclusion

HumanPrompt is a powerful framework that simplifies the integration and experimentation with various prompting methods. By following this guide, you can harness its capabilities for your research projects effectively!