Welcome to the world of Super-RAG, a super-performant RAG (Retrieval Augmented Generation) pipeline designed for AI applications! This guide will walk you through installing and using Super-RAG to facilitate efficient data processing in your AI projects. Let’s ignite your development journey!

Key Features

• Supports multiple document formats and vector databases.
• Provides a production-ready REST API.
• Customizable splitting/chunking methods.
• Options for encoding data using various encoding models (both proprietary and open source).
• Built-in code interpreter mode for computational question-answer scenarios.
• Session management enabled through unique IDs for caching purposes.

Cloud API

The easiest way to get started is to use our Cloud API. This API is free to use (within reasonable limits).

Installation

Follow these steps to get Super-RAG up and running:

1. Clone the repository:

   git clone https://github.com/superagent-ai/super-rag
   cd super-rag

2. Set up a virtual environment:
   • Using virtualenv:

     virtualenv env
     source env/bin/activate

   • Or using venv:

     python3 -m venv env
     source env/bin/activate

3. Install required packages:

   poetry install

4. Rename the .env.example file to .env and set your environment variables.
5. Run the server:

   uvicorn main:app --reload

Understanding the Setup with an Analogy

Imagine building a new gadget in your workshop. The repository is like the toolbox full of all the parts you need. When you clone the repository, it’s as if you are borrowing that toolbox to your workspace.

The virtual environment serves a very important purpose: it allows you to focus on just one project without interference from other tools or gadgets lying around your workspace. Think of it as creating a specialized area just for this gadget, ensuring everything operates smoothly together.

Installing the packages is like putting all the essential parts and tools you need into that area so you don’t have to keep searching for them. Finally, once you hit the run button (running the server), it’s time for your new gadget to power up and come to life!

How to Use Super-RAG

Super-RAG comes with a built-in REST API powered by FastAPI. Here is how you can interact with it:

Ingest Documents

To ingest documents, send a POST request to api/v1/ingest:

{
    "files": [
        {
            "name": "My file",  // Optional
            "url": "https://path-to-my-file.pdf",
            "metadata": {       // Optional
                "key": "value"
            }
        }
    ],
    "document_processor": {
        "encoder": {
            "dimensions": 384,
            "model_name": "embed-multilingual-light-v3.0",
            "provider": "cohere"
        },
        "unstructured": {
            "hi_res_model_name": "detectron2_onnx",
            "partition_strategy": "auto",
            "process_tables": false
        },
        "splitter": {
            "max_tokens": 400,
            "min_tokens": 30,
            "name": "semantic",
            "prefix_summary": true,
            "prefix_title": true,
            "rolling_window_size": 1
        }
    },
    "vector_database": {
        "type": "qdrant",
        "config": {
            "api_key": "YOUR API KEY",
            "host": "THE QDRANT HOST"
        },
        "index_name": "my_index",
        "webhook_url": "https://my-webhook-url"
    }
}

Query Documents

To query documents, send a POST request to api/v1/query:

{
    "input": "What is ReAct?",
    "vector_database": {
        "type": "qdrant",
        "config": {
            "api_key": "YOUR API KEY",
            "host": "THE QDRANT HOST"
        },
        "index_name": "YOUR INDEX"
    },
    "interpreter_mode": true,
    "encoder": {
        "provider": "cohere",
        "name": "embed-multilingual-light-v3.0",
        "dimensions": 384
    },
    "filter": {    // Optional
        // Each provider has its own filter schema
        "must": [
            {
                "key": "key",
                "match": {
                    "value": "value"
                }
            }
        ]
    },
    "exclude_fields": ["metadata"],
    "interpreter_mode": false,  // or true, for computational QA
    "session_id": "my_session_id"  // For caching purposes
}

Delete Document

To delete a document, send a POST request to api/v1/delete:

{
    "file_url": "A file url to delete",
    "vector_database": {
        "type": "qdrant",
        "config": {
            "api_key": "YOUR API KEY",
            "host": "THE QDRANT HOST"
        },
        "index_name": "my_index"
    }
}

Troubleshooting Common Issues

If you encounter any issues while using Super-RAG, consider the following troubleshooting steps:

• Ensure that your API keys and other configurations are correctly set in your `.env` file.
• Check if your virtual environment is activated before running the server.
• Verify the correct endpoint URLs when sending requests.
• Consult the server logs for any error messages that could guide troubleshooting.
• If issues persist, refer to the documentation or seek assistance from the community.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.