In the dynamic landscape of artificial intelligence, the task of image captioning has emerged as a pivotal function, enhancing the potential of text-to-image models. Introducing JoyCaption—a revolutionary Visual Language Model (VLM) that aims to democratize the availability of advanced image captioning tools. Designed from the ground up, JoyCaption is an open-access, uncensored model that caters to a diverse range of content creators, from safe-for-work (SFW) to not-safe-for-work (NSFW) domains. Let us delve into the distinguishing features that make JoyCaption a transformative force within the AI community.
Open and Accessible by Design
A hallmark of JoyCaption is its unwavering commitment to openness and accessibility. Released with open weights and devoid of restrictions, this model empowers anyone to utilize it for training diffusion models. This ethos resonates with the spirit of community-driven initiatives like bigASP, offering transparency and accessibility through comprehensive training scripts and insights into its developmental journey.
Uncensored and Inclusive
Breaking free from the constraints of heavily censored models such as ChatGPT, JoyCaption provides balanced coverage across SFW and NSFW concepts. This allows creators the liberty to explore a wide spectrum of themes and artistic styles. From digital art and photorealism to anime and furry content, JoyCaption celebrates diversity, accommodating various artistic preferences and cultural expressions.
Minimal Filtering Approach
With training on extensive datasets, JoyCaption captures the multifaceted nature of our world, delivering nuanced and comprehensive captions. While adhering to a strict policy against illegal content, its minimal filtering approach ensures the capability to process a broad range of images, augmenting its utility for diverse applications.
The Rationale Behind JoyCaption
The demand for automated descriptive captions has reached unprecedented levels. They facilitate the training and fine-tuning of diffusion models on a broader array of images, eliminating the necessity for pre-associated text or manual descriptions. JoyCaption addresses the shortcomings of existing solutions like ChatGPT, which are expensive and censored, and alternatives like CogVLM, which falter outside the SFW domain. By offering a free, unrestricted, and open model, JoyCaption fills a critical void in the community, performing comparably to leading models such as GPT-4o.
Initiating with JoyCaption
To begin your journey with JoyCaption, the model and its resources are available on [Github](https://github.com/fpgaminer/joycaption). The repository provides detailed instructions and examples to assist you in integrating JoyCaption into your projects. Here’s a succinct example of utilizing the model:
import torch import torchvision.transforms.functional as TVF from PIL import Image from transformers import AutoTokenizer, LlavaForConditionalGeneration IMAGE_PATH = "image.jpg" PROMPT = "Write a long descriptive caption for this image in a formal tone." MODEL_NAME = "John6666/llama-joycaption-alpha-two-hf-llava-nf4" # Load JoyCaption tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, use_fast=True) llava_model = LlavaForConditionalGeneration.from_pretrained(MODEL_NAME, torch_dtype=torch.bfloat16, device_map=0) llava_model.eval() with torch.no_grad(): # Load and preprocess image image = Image.open(IMAGE_PATH) if image.size != (384, 384): image = image.resize((384, 384), Image.LANCZOS) image = image.convert("RGB") pixel_values = TVF.pil_to_tensor(image) / 255.0 pixel_values = TVF.normalize(pixel_values, [0.5], [0.5]) pixel_values = pixel_values.to(torch.bfloat16).unsqueeze(0) # Build the conversation convo = [ {"role": "system", "content": "You are a helpful image captioner."}, {"role": "user", "content": PROMPT}, ] # Format and tokenize the conversation convo_string = tokenizer.apply_chat_template(convo, tokenize=False, add_generation_prompt=True) convo_tokens = tokenizer.encode(convo_string, add_special_tokens=False, truncation=False) input_tokens = [token if token != llava_model.config.image_token_index else [llava_model.config.image_token_index] * llava_model.config.image_seq_length for token in convo_tokens] input_ids = torch.tensor(input_tokens, dtype=torch.long).unsqueeze(0) attention_mask = torch.ones_like(input_ids) # Generate the caption generate_ids = llava_model.generate(input_ids=input_ids.to("cuda"), pixel_values=pixel_values.to("cuda"), attention_mask=attention_mask.to("cuda"), max_new_tokens=300, do_sample=True)[0] generate_ids = generate_ids[input_ids.shape[1]:] caption = tokenizer.decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False).strip() print(caption)
Conclusion
JoyCaption signifies a major advancement in the field of image captioning, offering a free, open, and uncensored solution for the AI community. By embracing inclusivity and minimizing filtering, it empowers creators to explore a vast array of content, thereby enhancing the quality and versatility of text-to-image models. Whether you are an artist, researcher, or developer, JoyCaption provides the tools necessary to actualize your creative vision.
