How to Install and Use ComfyUI ExLlama Nodes

Feb 3, 2023 | Data Science

If you’re looking to create a local text generator using the powerful ComfyUI with ExLlamaV2, you have landed on the right spot. This guide will walk you through the installation process and provide tips on how to effectively use it.

Installation Steps

To get started, you’ll need to clone the repository and install the necessary requirements. Here’s how to do it:

  • Navigate to your custom_nodes directory:
  • cd custom_nodes
  • Clone the ExLlama Nodes repository:
  • git clone https://github.com/Zuellni/ComfyUI-ExLlama-Nodes
  • Install the required packages:
  • pip install -r ComfyUI-ExLlama-Nodes/requirements.txt

For Windows users, you’ll need to install wheels for ExLlamaV2 and FlashAttention. Use the following commands:

  • Install ExLlamaV2:
  • pip install exllama-v2-X.X.X+cuXXX.torch2.X.X-cp3XX-cp3XX-win_amd64.whl
  • Install FlashAttention:
  • pip install flash_attn-X.X.X+cuXXX.torch2.X.X-cp3XX-cp3XX-win_amd64.whl

Usage Guidelines

Once you have set up everything, it’s time to put it to good use. The system supports EXL2, 4-bit GPTQ, and FP16 models, which can be downloaded from Hugging Face. Here’s how you can integrate a model:

  • Clone the model repository into your models directory:
  • cd models
    mkdir llm
    git clone https://huggingface.co/turboderp/Llama-3.1-8B-Instruct-exl2 -b 4.0bpw
  • You can also place models in a customized path by modifying the extra_model_paths.yaml file.

Understanding the Nodes

The ExLlama Nodes can be visualized as the individual gears in a beautifully crafted clock. Each gear has a specific role to play in the movement of the clock hands. Similarly, each node contributes to the overall function of generating text:

  • Loader: This node loads models from the llm directory.
  • Formatter: Prepares messages using the model’s chat template for a conversational experience.
  • Tokenizer: Focuses on processing input text, breaking it down into manageable parts.
  • Generator: The star of the show, this node generates text from the input provided.

Workflow Example

An example workflow can help you visualize how these nodes interact within ComfyUI. You can view the example image included in the repository:

Workflow Image

Troubleshooting Tips

While setting up or using ComfyUI with ExLlama Nodes, you may run into some common issues:

  • Ensure all installations were completed without errors. Re-run the installation commands if necessary.
  • Check if the model paths in extra_model_paths.yaml are correctly defined.
  • If loading the model takes too long or fails, reduce the cache_bits node’s value, which can help in managing VRAM usage.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.

Stay Informed with the Newest F(x) Insights and Blogs

Tech News and Blog Highlights, Straight to Your Inbox