PromptTools

🔧 Test and experiment with prompts, LLMs, and vector databases. 🔨

Welcome to PromptTools!

Created by Hegel AI, this repository offers a set of open-source, self-hostable tools that empower developers to experiment, test, and evaluate LLMs, vector databases, and prompts seamlessly. The goal is to create an intuitive environment for using familiar interfaces like code, notebooks, and a local playground.

Getting Started

In just a few lines of code, you can evaluate prompts and parameters across various models, such as OpenAI, Anthropic, or LLaMA models. Let’s break down how you can start testing your prompts:

from prompttools.experiment import OpenAIChatExperiment

messages = [
    {'role': 'user', 'content': 'Tell me a joke.'},
    {'role': 'user', 'content': 'Is 17077 a prime number?'},
]
models = ['gpt-3.5-turbo', 'gpt-4']
temperatures = [0.0]

openai_experiment = OpenAIChatExperiment(models, messages, temperature=temperatures)
openai_experiment.run()
openai_experiment.visualize()

Think of this code like a recipe. You have ingredients (models, messages, temperatures) that you combine to create a delicious dish (experiment). Each ingredient has its unique role: messages are the questions you ask, models represent the chefs preparing your answers, and temperatures determine the spice level—how creative or straightforward the response should be.

Installation

To install PromptTools, you simply need to use pip:

pip install prompttools

After installing, you can run a simple example locally:

git clone https://github.com/hegelaiprompttools.git
cd prompttools
jupyter notebook examples/notebooks/OpenAIChatExperiment.ipynb

You can also try this notebook in Google Colab.

Using the Playground Interface

If you’d like to interact with PromptTools using our playground interface, you can launch it with the following commands:

pip install notebook  # Install Jupyter Notebook if not yet installed
pip install prompttools
git clone https://github.com/hegelaiprompttools.git
cd prompttools
streamlit run prompttools/playground/playground.py

You can also access a hosted version of the playground on the Streamlit Community Cloud (Note: The hosted version does not support LlamaCpp).

Supported Integrations

PromptTools supports a variety of APIs for experiments, including:

• OpenAI (Completion, ChatCompletion, Fine-tuned models)
• LLaMA.Cpp (LLaMA 1, LLaMA 2)
• HuggingFace (Hub API, Inference Endpoints)
• Anthropic
• Mistral AI
• Google Gemini
• Google PaLM (legacy)
• Google Vertex AI
• Azure OpenAI Service
• Replicate
• Ollama (In Progress)

And for vector databases, we support:

• Chroma
• Weaviate
• Qdrant
• LanceDB
• Milvus (Exploratory)
• Pinecone
• Epsilla (In Progress)

Troubleshooting

Here are some common issues and solutions to help you along the way:

• Issue: The prompts don’t seem to deliver consistent responses.
• Solution: Experiment with different models or modify the “temperature” setting to adjust creativity.
• Issue: Installation errors while using pip.
• Solution: Ensure you have the necessary permissions and that your Python environment is set up correctly.
• If you encounter any other issues, feel free to reach out for support in our Discord channel.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Frequently Asked Questions (FAQs)

• Will this library forward my LLM calls to a server? No, the source code will be executed on your machine without any forwarding.
• Does prompttools store my API keys or LLM inputs and outputs to a server? No, all data remains local, and we do not collect any PII.
• How do I persist my results? You can export your experiments using methods like to_csv, to_json, or to_mongo_db.

Contributing

We welcome contributions! Check out our Help Wanted issues to get involved.

Conclusion

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.