Welcome to the magical world of graphics, where the potential of your designs can be unleashed with MatForger! This generative diffusion model is tailored for creating Physically Based Rendering (PBR) materials. Whether you are a designer, artist, or developer, MatForger equips you with the tools to bring your artistic visions to life.

Model Overview

Inspired by the MatFuse model and trained on the comprehensive MatSynth dataset, MatForger innovates texture synthesis by implementing a noise rolling technique derived from ControlMat. This allows for seamless tileable maps, making it ideal for various material designs.

Features of MatForger

• High-Quality PBR Material Generation: Produces finely detailed materials for numerous applications.
• Tileable Textures: Ensures that textures can be seamlessly repeated across larger scenes.
• Versatile Outputs: Generates multiple texture maps, including basecolor, normal, height, roughness, and metallic.
• Text and Image Conditioning: Allows you to customize material generation with text prompts or images, enabling creative freedom.

Getting Started with MatForger

Using MatForger requires a custom pipeline, which can be easily set up in Diffusers. Here’s a simplified analogy to help understand the code structure:

Imagine you are a chef preparing a multi-course meal. Each step in your recipe corresponds to the lines of code in MatForger. First, you gather your ingredients (loading the model), then you set the kitchen temperature (selecting the device), followed by preparing your main dish (prompting with image or text), and finally, you set the table for your guests (outputting the texture maps). Just like cooking, following the right steps will lead you to a delightful result!

import torch
from PIL import Image
from diffusers import DiffusionPipeline

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
pipe = DiffusionPipeline.from_pretrained("gvecchio/MatForger", trust_remote_code=True)
pipe.enable_vae_tiling()
pipe.enable_freeu(s1=0.9, s2=0.2, b1=1.1, b2=1.2)
pipe.to(device)

# Load and generate materials based on an image
prompt = Image.open("bricks.png")
image = pipe(prompt, guidance_scale=6.0, height=512, width=512, tileable=True, patched=False, num_inference_steps=25).images[0]

# Alternatively, you can use text prompts
prompt = "terracotta brick wall"
image = pipe(prompt, guidance_scale=6.0, height=512, width=512, tileable=True, patched=False, num_inference_steps=25).images[0]

# Extracting texture maps from the generated image
basecolor = image.basecolor
normal = image.normal
height = image.height
roughness = image.roughness
metallic = image.metallic

Troubleshooting and Tips

As with any journey in the world of programming, you may encounter some bumps along the way. Here are a few troubleshooting tips:

• Quality Issues: If the generated materials do not meet your expectations, be patient and experiment with different prompts and parameters.
• Resolution Artifacts: Since MatForger was trained at a resolution of 256×256, higher resolutions may introduce artifacts. Stick to this resolution for best results.
• Feedback and Contributions: We welcome suggestions for new features or feedback! Your insights are vital for developing MatForger further.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Future Enhancements

As we continue to enhance MatForger’s capabilities, several exciting features are in the pipeline:

• Opacity Maps: Designing materials with transparency.
• Material Inpainting: Correcting or enhancing materials within the generated textures.
• Sketch-Based Material Generation: Creating textures from simple sketches.
• Improved Color Palette Control: Achieving precise color matching.
• Photo-Based Material Estimation: Mimicking textures from real-world images.

Join the Development Journey

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.