Welcome to the guide on how to set up and use RainNet, an innovative and effective approach to image harmonization that harnesses the power of Region-aware Adaptive Instance Normalization (RAIN). In this article, we will walk you through the preparation, usage, and evaluation steps to make the most out of this PyTorch implementation.

Table of Contents

• Introduction
• Preparation
• Usage
• Results
• Citation
• Acknowledgment

Introduction

This project conceptualizes image harmonization as a style transfer problem. The RAIN module formulates the visual style from the background and adapts it for the foreground. With proper implementation, this module can be seamlessly integrated into existing frameworks for substantial improvements in output quality. Extensive experiments have demonstrated the superiority of this method on benchmark datasets.

Preparation

Step 1: Clone the Repository

First, clone the RainNet repository to your local machine:

git clone https://github.com/junlee/RainNet
cd RainNet

Step 2: Install Requirements

RainNet is compatible with both Linux and Windows, though Linux is preferred for better compatibility. Ensure you have Python 3.6 installed and PyTorch version 1.4.0 or higher. To install the required packages, use:

pip3 install -r requirements.txt

Or via conda:

conda create -n rainnet python=3.6
conda activate rainnet
pip install -r requirements.txt

Step 3: Prepare the Data

Download the iHarmony4 dataset and extract the images. Due to their size, we recommend resizing the images (e.g., 512×512 or 256×256). Use the provided script:

python datapreprocess_iharmony4.py --dir_iharmony4 DIR_of_iHarmony4 --save_dir SAVE_DIR --image_size IMAGE_SIZE

If you’d like to maintain the aspect ratio, you can include the flag --keep_aspect_ratio in your command.

Step 4: Download Pre-trained Model

To benefit from a pre-trained model, download it from Google Drive or Baidu Drive (Code: 3qjk). Place net_G_last.pth in the directory checkpoints/experiment_train.

Usage

Step 1: Evaluation

To evaluate your model with the dataset, run:

python evaluate.py --dataset_root DATA_DIR --save_dir evaluated --batch_size 16 --device cuda

You can add --store_image to save the harmonized images to your specified directory.

Step 2: Testing with Your Own Examples

For testing, edit test.py to specify your image paths. For instance:

comp_path = ['examples1.png', 'examples2.png']
mask_path = ['examples1-mask.png', 'examples1-mask.png']
real_path = ['examples1-gt.png', 'examples2-gt.png']

Set real_path to None if no ground truth images are available.

Step 3: Training Your Own Model

To train your own model, update the command arguments in scripts/train.sh and execute:

bash scripts/train.sh

Results

RainNet has showcased remarkable results and improvements in image harmonization, as illustrated in the following comparisons:

Troubleshooting

If you encounter issues during setup or execution, here are some common troubleshooting tips:

• Ensure you have the correct version of Python and PyTorch installed.
• Double-check the file paths for images and model weights.
• If running on Windows, consider using Windows Subsystem for Linux (WSL) for better compatibility.
• For detailed community discussions or questions, refer to the RainNet repository on GitHub.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Citation

If you utilize this project or find it beneficial, kindly cite our paper:

@inproceedings{ling2021region,
  title={Region-aware Adaptive Instance Normalization for Image Harmonization},
  author={Ling, Jun and Xue, Han and Song, Li and Xie, Rong and Gu, Xiao},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={9361--9370},
  year={2021}
}

Acknowledgment

We acknowledge the contributions of the DoveNet and pix2pix repositories for various data modules and model functions employed in this code.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.