In the age of information, having a Local Model (LLM) that can scour the web for the latest information is an invaluable asset. This blog will guide you through the process of enhancing your local LLM’s capabilities to perform web searches effectively.

Table of Contents

• Installation
• Usage
• Search Backends
• Keyword Retrievers
• Recommended Models

Installation

Setting up this extension involves a few steps:

1. Navigate to the Session tab of the web UI and use the Install or update an extension option to download the latest code.
2. Install dependencies:
• The easy way: Run the update_wizard script inside the text-generation-webui folder. Choose the option to install/update extension requirements.
• The safe way: Manually update the conda environment using the command:

  conda env update -p path_to_your_environment --file environment.yml

  or, for your own environment:

  conda env update -n name_of_your_environment --file environment.yml

3. Launch the Web UI with:

   python server.py --extension LLM_Web_search

   Confirm that the extension was loaded by checking for a new tab titled LLM Web Search in the web UI.

Usage

To use this extension effectively, follow these steps:

1. Load your model.
2. Load a matching instruction template.
3. Navigate to the LLM Web search tab.
4. Load a custom system message prompt.
5. Make sure the query in the system message matches the current Search command regex string.
6. Select an appropriate hyperparameter generation preset.
7. Choose between chat-instruct or instruct mode to begin chatting.

Using a Custom Regular Expression

The default regex is regexpSearch_web((.*)). This regex extracts search queries from the model’s output. A more flexible and complex regex example is:

regexp[Ss]earch_web((?:[])(.*)(?:[]))

Tools like Debuggex can help in refining your custom regex patterns.

Reading Web Pages

Extraction of full text from web pages is possible with the regex: regexpOpen_url((.*)). However, be mindful that this may lead to exceeding your LLM’s context length.

Search Backends

DuckDuckGo

By default, DuckDuckGo serves as your web search backend.

SearXNG

You can opt for SearXNG by pasting your local or remote SearXNG instance URL into the appropriate settings field. This must support JSON results.

Keyword Retrievers

Okapi BM25

This extension comes pre-loaded with Okapi BM25, a popular tool for keyword-based document retrieval.

SPLADE

If you have VRAM available, consider using SPLADE. It utilizes query expansion for added context but may be slower than BM25. Install it by running:

pip install qdrant-client

Remember, adjusting the SPLADE batch size can enhance performance but will use more VRAM.

Recommended Models

For users with ≤ 12 GB VRAM, Llama-3-8B-instruct is highly recommended. You can find suitable instruction templates in the extensions folder.

Troubleshooting

If you encounter challenges, consider the following tips:

• Ensure all dependencies are installed correctly, following the installation instructions closely.
• If regex fails, double-check that the search command and regex match precisely.
• Monitor memory usage if running into performance issues, especially with SPLADE.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

With the ability to search the web, your local LLM transforms from a static tool into a dynamic information seeker, enhancing its effectiveness. At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.