In this article, we’ll guide you through setting up a local Large Language Model (LLM) that employs Retrieval-Augmented Generation (RAG). Picture this project as a wizard experimenting with powerful spells to conjure up answers from PDFs by harnessing the might of Ollama and other tools. Let’s dive in and explore how you can create your own enchanting AI assistant!

Requirements

• Ollama version 0.1.26 or higher

Setup Steps

1. Clone the repository to your local machine.
2. Create a Python virtual environment by running:

   python3 -m venv .venv

3. Activate the virtual environment:
   • On Unix or MacOS:

     source .venv/bin/activate

   • On Windows:

     .venv\Scripts\activate

4. Install the required Python packages by running:

   pip install -r requirements.txt

Running the Project

When you first run the project, it will fetch the necessary models from Ollama for the LLM and embeddings. This process is akin to a wizard gathering ingredients for a powerful potion—it may take some time based on your internet speed.

1. Ensure your virtual environment is activated.
2. Run the main script with:

   python app.py -m model_name -p path_to_documents

   This allows you to specify a model and the path to your documents. If you choose not to specify a model, it defaults to mistral. The document path defaults to a sample location called “Research”.
3. You can optionally specify the embedding model with:

   -e embedding_model_name

   If you leave this unspecified, it defaults to nomic-embed-text.

This process loads your PDFs and Markdown files, generates embeddings, queries the collection, and delivers answers as outlined in app.py.

Running the Streamlit UI

1. Ensure your virtual environment is activated.
2. Navigate to the directory containing ui.py.
3. Run the Streamlit application by executing:

   streamlit run ui.py

This command starts a local web server and opens a new tab in your default web browser, allowing you to interact with the application through a user-friendly interface. The Streamlit UI makes it significantly easier to select models, choose folder paths, and communicate with the RAG chatbot.

Troubleshooting Tips

If you encounter issues during setup or execution, consider the following:

• Ensure that your Python version is compatible with the packages you are trying to install.
• Verify your internet connection; downloading models can be interrupted by unstable connectivity.
• If you receive errors regarding missing packages, re-run the installation command specified in the setup section.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Technologies Used

• Langchain: A Python library tailored for working with Large Language Models.
• Ollama: A robust platform for local LLM usage.
• Chroma: A vector database for storing and retrieving embeddings.
• PyPDF: A Python library for reading and manipulating PDF files.
• Streamlit: A web framework designed for creating interactive applications in machine learning and data science.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.