With the recent advancements in AI technology, the llama.cpp team has introduced GGUF as a new and improved alternative to the previously used GGML. This blog will guide you step-by-step on transitioning to GGUF, ensuring your projects stay up-to-date and efficient.

Understanding GGUF

GGUF, launched on August 21, 2023, is the fresh breeze meant to take over GGML, which is no longer supported. The primary branch will persist with the deprecated variant, but for full support and the latest features, switching to the gguf-fix branch is essential.

Setup Instructions

• Using Docker:

  Setting up GGUF using Docker can be done rapidly. Here’s how:

  docker pull su77ungrcasalioy:stable
  docker run -it -p 8501:8501 --shm-size=16gb su77ungrcasalioy:stable /bin/bash

  If you’re aiming for GPU support, use:

  docker pull su77ungrcasalioy:gpu

  You’re set to start ingesting your dataset!

• Building from Source:

  For those who prefer building from source, follow these steps:

  python -m pip install poetry
  python -m poetry config virtualenvs.in-project true
  python -m poetry install
  source .venv/bin/activate
  python -m pip install --force streamlit sentence_transformers

Ingesting Your Own Dataset

Once setup is complete, you can ingest multiple document types including .txt, .pdf, .csv, and more by executing:

python casalioy/ingest.py [path_to_your_data_directory]

If you wish to purge existing data and start fresh, add the y flag:

python casalioy/ingest.py [path_to_your_data_directory] -y

Asking Questions Locally

To interrogate your documents locally, use the command below. The prompt will wait for your input:

python casalioy/startLLM.py

After entering your query, the system may take a short 20-30 seconds to generate responses based on the context retrieved from your documents.

New GUI Feature

The newly added GUI interface allows for an interactive experience. You can run it with the following command:

streamlit run casalioy/gui.py

Troubleshooting

If you encounter issues during the installation or operational phase, here are some suggestions:

• Ensure Docker is installed and running smoothly.
• Check your environment variables in .env for accuracy.
• Verify your document formats are supported for ingestion.
• If Streamlit throws errors, confirm that Python and Streamlit versions are compatible.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.

Conclusion

Transitioning to GGUF is not just a necessity but an opportunity to leverage improved functionalities in your LLMs. Stay informed and update your systems accordingly to make the most of this innovative framework!