Are you ready to turn your sketches into beautiful, high-fidelity images? With the recent advancements in generative models, specifically the Paint-by-Sketch framework, you can unleash your artistic imagination by combining sketches with reference images. This step-by-step guide will help you navigate through the setup and utilization of this powerful tool.

What is Paint-by-Sketch?

Paint-by-Sketch is an innovative image composition model that combines the flexibility of sketches with reference images to generate detailed artworks. Imagine having the ability to create a masterpiece by simply outlining your ideas and providing a reference to enrich the details. This model utilizes edge-level control to allow for editable parts in your images, promoting user-driven modifications of scenes.

How to Set Up Paint-by-Sketch

Getting started involves a few key steps including setting up the environment and downloading model checkpoints. Here’s how you can do it:

Step 1: Environment Setup

• First, create an environment using Conda with the provided command:

conda env create -f environment.yaml

• Activate the environment:

conda activate paint_sketch

• Install the required dependencies:

pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113

• Continue by installing additional packages:

pip install opencv-python==4.6.0.66 opencv-python-headless==4.6.0.66 matplotlib==3.2.2 streamlit==1.14.1 streamlit-drawable-canvas==0.9.2

• Lastly, install the CLIP package:

pip install git+https://github.com/openai/CLIP.git

Step 2: Checkpoint Download

Download the pre-trained model checkpoints from Google Drive.

Place the downloaded checkpoint file in the correct directory structure as follows:

Paint-by-Sketch/
    └── pretrained_models/
        └── model-modified-12channel.ckpt/
            └── models/
                └── Cartoon_v1_aesthetic/
                    ...

Data Preparation Process

To prepare your data, execute the sketch extraction by running:

bash preprocess_data/run_preprocess.sh path_to_image_root gpu_id

For example:

bash preprocess_data/run_preprocess.sh home/nas2_userF/kangyeol/Project/webtoon2022/Paint-by-Sketch/samples 7

Training the Model

To train your model, use:

bash cartoon_train.sh gpu_ids path_to_logdir path_to_config

For example:

bash cartoon_train.sh 0,1 models/test config/v1_aesthetic_sketch_image.yaml

Ensure that the number of GPU IDs matches the GPUs specified in the lightning.trainer.yaml file.

Running the Demo

Once everything is set up, you can run the Streamlit server to demonstrate Paint-by-Sketch by using:

bash streamlit run demo/app.py --server.port=8507 --server.fileWatcherType none

Follow these steps in the demo:

• Upload the source image.
• Draw masks and sketches in separate canvases.
• Upload and read a reference image, and crop it as needed.
• Finally, perform inference and export your finalized images!

Troubleshooting Common Issues

If you experience misalignment with your drawings, double-check that your screen size is adequate for the canvas. If the canvas size changes due to a small screen, it may lead to misalignment of your drawn masks and sketches.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

Paint-by-Sketch provides a unique opportunity to interact with image generation technology, allowing you to express your creativity through sketches. By following this guide, you are now equipped to begin your artistic journey using this model.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.