How to Use Flux.1-dev: Surface Normals ControlNet

Welcome to the fascinating world of image transformation with the Flux.1-dev ControlNet for Surface Normals! In this guide, we will walk you through the steps required to use this powerful model, developed by the Jasper research team. Let’s dive in!

What You’ll Need

• A working Python environment
• The diffusers library
• A GPU setup with CUDA

Getting Started

Before you start, ensure you have the diffusers library installed. If you haven’t done so, you can install it using pip:

pip install diffusers

Loading the Model

The model is built to handle images, particularly for generating surface normals. Let’s break down the code necessary to bring this model to life, just like crafting a fine dish where every ingredient plays a vital role.

import torch
from diffusers.utils import load_image
from diffusers import FluxControlNetModel
from diffusers.pipelines import FluxControlNetPipeline

# Load pipeline
controlnet = FluxControlNetModel.from_pretrained(
    "jasperai/Flux.1-dev-Controlnet-Surface-Normals",
    torch_dtype=torch.bfloat16
)
pipe = FluxControlNetPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    controlnet=controlnet,
    torch_dtype=torch.bfloat16
)
pipe.to("cuda")

Understanding the Code with an Analogy

Imagine you’re making a custom pizza. You have your base (the controlnet), the toppings (the pipes), and the cooking method (using CUDA). Each part must come together seamlessly. First, you gather the ingredients (importing libraries), then you prepare the dough (loading controlnet), and finally, you pay attention to the oven temperature (using pipe.to(“cuda”)) to ensure your pizza bakes perfectly!

Loading a Control Image

After loading the model, you need to obtain a control image. This is the image that guides the transformation process:

control_image = load_image(
    "https://huggingface.co/jasperai/Flux.1-dev-Controlnet-Surface-Normals/resolve/main/examples/surface.jpg"
)

Generating the Image

Now it’s time to generate your image with a specific prompt!

prompt = "a man showing stop sign in front of window"
image = pipe(
    prompt,
    control_image=control_image,
    controlnet_conditioning_scale=0.6,
    num_inference_steps=28,
    guidance_scale=3.5,
    height=control_image.size[1],
    width=control_image.size[0]
).images[0]

Visualizing the Output

Your generated image can be displayed like a beautifully decorated pizza ready to serve:

Additional Steps: Computing the Conditioning Map

If you need to compute the conditioning map for more intricate transformations, you can do so using the NormalBaeDetector. This step is akin to adding secret spices to enhance your pizza flavor:

from controlnet_aux import NormalBaeDetector
from diffusers.utils import load_image

normal_bae = NormalBaeDetector.from_pretrained("lllyasviel/Annotators")
normal_bae.to("cuda")

# Load an image
im = load_image(
    "https://huggingface.co/jasperai/Flux.1-dev-Controlnet-Surface-Normals/resolve/main/examples/output.jpg"
)
surface = normal_bae(im)

Licensing Information

This model falls under the Flux.1-dev model license.

Troubleshooting Tips

If you encounter any issues while using the model, consider the following troubleshooting ideas:

• Ensure you have the correct version of the diffusers library installed.
• Check if your CUDA setup is properly configured and your GPU is recognized.
• Review the accuracy of your image URL links.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.