Welcome to our comprehensive guide on using Paddler, the robust open-source tool designed to enhance the performance and efficiency of servers running llama.cpp. In this article, we will dive into what Paddler is, why it’s essential, and how you can set it up properly.

Why Paddler?

Traditional load balancing strategies such as round robin or least connections fall short when handling llama.cpp servers. This is primarily due to their reliance on continuous batching algorithms and custom configurations to manage multiple requests. Paddler is tailored specifically to interact seamlessly with llama.cpp’s unique feature of slots — predefined memory slices within the server responsible for managing individual requests. Think of slots as dinner tables where each table can host multiple diners (requests) at the same time. Efficiently distributing requests to these tables is crucial for a smooth dining experience.

To learn more about slots, visit the llama.cpp server documentation.

Key Features

• Agent monitoring of individual llama.cpp instance slots.
• Dynamic addition/removal of llama.cpp servers, supporting autoscaling.
• Request buffering for scaling from zero hosts.
• Integration with the StatsD protocol.
• Built-in dashboard and AWS compatibility.

How Paddler Works

Setting up Paddler involves registering your llama.cpp instances so that agents can report their slot status to the load balancer. Once the agents are in place, they operate in a cycle akin to how a waiter checks in with the kitchen and fills orders.

Here’s how the interaction works:

Agent: Hey, are you alive?
llama.cpp: Yes, this is my slots status
Agent: llama.cpp is still working
Load Balancer: I have a request for you to handle

Usage Instructions

Installation

To get started, download the latest release for your corresponding OS—be it Linux, Mac, or Windows—from the releases page. For Linux users wanting system-wide accessibility, rename the downloaded executable to /usr/bin/paddler or /usr/local/bin/paddler.

Running Agents

The agents need to be installed on the same host as your llama.cpp instance. Each agent requires the following configuration:

1. external-*: Connection details for the load balancer to connect to the llama.cpp instance.
2. local-*: Connection details for the agent to connect to the llama.cpp instance.
3. management-*: Information on where the agent reports the slots’ status.

To start a Paddler agent, run the following command (make sure to replace the placeholder addresses with your actual server addresses):

paddler agent \
    --external-llamacpp-host 127.0.0.1 \
    --external-llamacpp-port 8088 \
    --local-llamacpp-host 127.0.0.1 \
    --local-llamacpp-port 8088 \
    --management-host 127.0.0.1 \
    --management-port 8085

Naming the Agents

With Paddler Version 0.6.0 and above, you can assign a custom name to each agent using the --name flag, which will display in the management dashboard.

Starting the Load Balancer

Once the agents are running, you can start the load balancer, which collects data from agents and exposes a reverse proxy to external requests:

paddler balancer \
    --management-host 127.0.0.1 \
    --management-port 8085 \
    --reverseproxy-host 192.168.2.10 \
    --reverseproxy-port 8080

Enabling the Dashboard

To view agent status, you can enable the dashboard with the --management-dashboard-enable=true flag. After activating it, you can access the dashboard at the management server address under the dashboard path.

Troubleshooting Tips

• If agents are not reporting their slots correctly, ensure that the connection details are accurate and reachable.
• Check for network issues between the agents and load balancer.
• If the dashboard does not display data, confirm that the management flags are set correctly.
• For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.

Final Thoughts

With Paddler, you’re equipped to optimize your llama.cpp servers efficiently. Embrace the power of stateful load balancing to enhance your application performance and scalability. As developments in AI and machine learning continue to surge, the importance of the right infrastructure cannot be underestimated.