Welcome to your go-to guide for successfully implementing Relational Knowledge Distillation (RKD), a cutting-edge technique from the CVPR 2019 conference.

Getting Started

This repository contains the source code needed for experiments in metric learning using relational knowledge distillation. We’ll walk you through the essential steps to get started, including installation and how to run various commands.

Quick Start Guide

• Firstly, run the following command to see all available options:

bash python run.py --help

• Next, run another help command to explore more options for distillation:

python run_distill.py --help

Training the Teacher Network

Now, let’s train a teacher embedding network using ResNet50 with a dimension of 512, utilizing triplet loss:

python run.py --mode train                 --dataset cub200                --base resnet50                --sample distance                 --margin 0.2                 --embedding_size 512                --save_dir teacher

Evaluating the Teacher Network

Once your model has been trained, it’s time to evaluate the teacher embedding network:

python run.py --mode eval                 --dataset cub200                --base resnet50                --embedding_size 512                --load teacherbest.pth

Distilling to the Student Network

Next, we’ll distill the learned knowledge from the teacher embedding network to a student embedding network (ResNet18):

python run_distill.py --dataset cub200                       --base resnet18                       --embedding_size 64                       --l2normalize false                       --teacher_base resnet50                       --teacher_embedding_size 512                       --teacher_load teacherbest.pth                       --dist_ratio 1                        --angle_ratio 2                       --save_dir student

Evaluating the Student Network

Finally, to evaluate the trained student network, execute:

python run.py --mode eval                 --dataset cub200                --base resnet18                --l2normalize false                --embedding_size 64                --load studentbest.pth

Dependencies

Before diving in, ensure you have the following dependencies installed:

• Python 3.6
• Pytorch 1.0
• tqdm: pip install tqdm
• h5py: pip install h5py
• scipy: pip install scipy

Important Notes

• The hyper-parameters used for experiments can be found in the scripts in examples.
• A heavy teacher network (ResNet50 with 512 dimensions) requires over 12GB of GPU memory when using a batch size of 128. If you encounter memory issues, consider reducing the batch size. Note that the experiments in the paper were conducted on a P40 with 24GB of GPU memory.

Troubleshooting

If you run into issues, here are a few ideas to troubleshoot:

• Ensure that all dependencies are installed correctly and are compatible with Python 3.6.
• Check if your GPU has sufficient memory. You might have to adjust the batch size accordingly.
• If there are issues with loading the trained model, verify the model path and ensure that the model files are in the correct location.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.