How to Merge FLUX Models: A Step-by-Step Guide

In the world of AI, merging different model parameters can create more efficient and powerful solutions. The FLUX models from black-forest-labs serve as great examples of how to achieve this. In this blog, we’ll guide you through the process of merging the parameters from the FLUX.1-dev and FLUX.1-schnell models.

Understanding the Merge Process

Imagine you’re trying to combine flavors in cooking; each ingredient has its unique profile, and when used together, they can create a new, harmonious flavor. Similarly, merging models involves combining parameters from two separate models to create a new, more efficient model. The resulting model can take advantage of the strengths of both original models while mitigating weaknesses.

Step-by-Step Guide to Merging Models

1. Install Required Libraries: Ensure you have the necessary libraries installed in your Python environment. You’ll need diffusers, huggingface_hub, and torch.
2. Load Models: Use the following code to load the two models:

from diffusers import FluxTransformer2DModel
from huggingface_hub import snapshot_download
from accelerate import init_empty_weights

with init_empty_weights():
    config = FluxTransformer2DModel.load_config("black-forest-labs/FLUX.1-dev", subfolder="transformer")
    model = FluxTransformer2DModel.from_config(config)

3. Download Checkpoints: Let’s download the model checkpoints with the given codes.

dev_ckpt = snapshot_download(repo_id="black-forest-labs/FLUX.1-dev", allow_patterns="transformer*")
schnell_ckpt = snapshot_download(repo_id="black-forest-labs/FLUX.1-schnell", allow_patterns="transformer*")

4. Merge Parameters: The next step involves merging the parameters from both models:

merged_state_dict = {}
guidance_state_dict = {}

for i in range(len(dev_shards)):
    state_dict_dev_temp = safetensors.torch.load_file(dev_shards[i])
    state_dict_schnell_temp = safetensors.torch.load_file(schnell_shards[i])
    keys = list(state_dict_dev_temp.keys())
    
    for k in keys:
        if "guidance" not in k:
            merged_state_dict[k] = (state_dict_dev_temp.pop(k) + state_dict_schnell_temp.pop(k)) / 2
        else:
            guidance_state_dict[k] = state_dict_dev_temp.pop(k)
    
    if len(state_dict_dev_temp) > 0:
        raise ValueError(f"There should not be any residue but got: {list(state_dict_dev_temp.keys())}.")
    if len(state_dict_schnell_temp) > 0:
        raise ValueError(f"There should not be any residue but got: {list(state_dict_dev_temp.keys())}.")

merged_state_dict.update(guidance_state_dict)

5. Save the Merged Model: Finally, save the new merged model with the following code:

load_model_dict_into_meta(model, merged_state_dict)
model.to(torch.bfloat16).save_pretrained("merged-flux")

Inference with the Merged Model

Once the models are merged and saved, you can use them for inference. Here’s a sample code snippet on how to do so:

from diffusers import FluxPipeline

pipeline = FluxPipeline.from_pretrained("sayakpaul/FLUX.1-merged", torch_dtype=torch.bfloat16).to("cuda")
image = pipeline(prompt="a tiny astronaut hatching from an egg on the moon", 
                 guidance_scale=3.5, 
                 num_inference_steps=4, 
                 height=880, 
                 width=1184, 
                 max_sequence_length=512, 
                 generator=torch.manual_seed(0)).images[0]

image.save("merged_flux.png")

Troubleshooting Tips

If you encounter issues while merging or running the model, consider the following troubleshooting ideas:

• Check that all dependencies are up to date. Sometimes libraries may change, causing compatibility issues.
• Ensure that the model paths in the code are correct. An incorrect path can lead to errors while loading the models.
• Review the error messages closely. They often provide clues about what went wrong.
• If you run into memory issues, consider reducing batch sizes or optimizing the model loading methods.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

With this guide, you should now have a clearer understanding of how to merge FLUX models effectively. By leveraging the strengths of both models, you can create more powerful AI applications.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.