Welcome to the world of large language models (LLMs) tailored for Rockchip RK3588! In this guide, we’ll explore how to leverage the ezrkllm-collection for your projects. This collection comes equipped with a selection of converted models optimized specifically for SBCs like Orange Pi 5, NanoPi R6, and Radxa Rock 5.
Getting Started
Before diving into the installation and usage of these models, it’s essential to know that they’ll be running on Rockchip’s efficient RK3588 NPU. Below are the steps to get you set up and running with LLMs.
Available Large Language Models (LLMs)
Here are the LLMs currently available in the ezrkllm-collection:
- Qwen Chat – 1.8B parameters – Link
- Gemma – 2B parameters – Link
- Microsoft Phi-2 – 2.7B parameters – Link
- Microsoft Phi-3 Mini – 3.8B parameters – Link
- Llama 2 7B – 7B parameters – Link
- Llama 2 13B – 13B parameters – Link
- TinyLlama v1 – 1.1B parameters – Link
- Qwen 1.5 Chat – 4B parameters – Link
- Qwen 2 – 1.5B parameters – Link
Installation Steps
To get started, you’ll need to download one of the models. Follow these steps:
- Use the following command to clone the desired model:
git clone LINK_FROM_PREVIOUS_TABLE_HERE - In some cases, you may need to execute:
git lfs pull - If the cloning process is too slow, try the following command:
GIT_LFS_SKIP_SMUDGE=1 git clone LINK_FROM_PREVIOUS_TABLE_HERE - Then, once inside the cloned folder, execute:
git lfs pull
Understanding RKLLM Parameters
It’s crucial to note that RK3588 supports only w8a8 quantization, which is used for optimal model performance. Models in this toolkit are optimized to strike the right balance of size and efficiency.
Analogy to Understand Model Sizing and Memory Usage
Imagine you’re planning a dinner party, and each guest needs a certain amount of space on the table to enjoy their meal. The “size” of each model can be thought of as the number of guests attending, while your RAM is the size of the table. For every guest (or model), you’ll need enough space to fit them comfortably – about 1.5 to 3 times the size of the guest (or model). If your table (or RAM) is too small, some guests may have to sit on the floor!
Troubleshooting Common Issues
If you encounter problems during installation or usage, consider the following troubleshooting tips:
- Make sure that the correct Rockchip RK3588 software is installed and configured properly.
- Check log outputs for specific error messages that can guide your next steps.
- Ensure adequate RAM and swap space is available as per the requirements mentioned earlier.
For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.
Future Directions
Looking ahead, there are plans to convert more LLMs and add support for additional Rockchip SoCs. This means more options for you to choose from in the future!
At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.
Happy coding and good luck on your journey to harness the power of LLMs with your Rockchip RK3588!