Welcome to your guide on using the Whisper automatic speech recognition (ASR) model, a powerful tool designed to transcribe and translate spoken languages seamlessly. In this article, we will walk you through the steps to set up and utilize the Whisper model effectively.

What is Whisper?

Whisper is a pre-trained model specializing in automatic speech recognition and speech translation, designed to handle multiple languages with ease. Think of it as a multilingual translator that can convert spoken words into written text.

Setting Up the Whisper ASR Model

To get started, you will need to have the necessary libraries installed and follow these steps:

• Ensure you have Whisper model ready.
• Set up your Python environment with the appropriate libraries.

Requirements

You will require:

• PyTorch
• Transformers Library
• Librosa for audio processing

Implementing the Code

Here’s a simple analogy to help you understand the implementation code. Imagine the Whisper model is like a chef preparing a gourmet meal:

• The WhisperProcessor is your kitchen chef’s assistant, helping sort out the important ingredients (audio features).
• The WhisperForConditionalGeneration is the main chef who puts everything together to create our desired dishes (transcripts).
• Just as chefs require specific instructions to prepare each dish, we use input_features to guide our Whisper model.

import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration

SAMPLING_RATE = 16000
has_cuda = torch.cuda.is_available()
model_path = "ivrit-ai/whisper-v2-d3-e3"
model = WhisperForConditionalGeneration.from_pretrained(model_path)

if has_cuda:
    model.to("cuda:0")
    
processor = WhisperProcessor.from_pretrained(model_path)

audio_resample = librosa.resample(entry['audio']['array'], orig_sr=entry['audio']['sampling_rate'], target_sr=SAMPLING_RATE)
input_features = processor(audio_resample, sampling_rate=SAMPLING_RATE, return_tensors="pt").input_features

if has_cuda:
    input_features = input_features.to("cuda:0")

predicted_ids = model.generate(input_features, language='he', num_beams=5)
transcript = processor.batch_decode(predicted_ids, skip_special_tokens=True)

print(f"Transcript: {transcript[0]}")

Evaluating and Extending Transcriptions

The Whisper model can transcribe audio samples of up to 30 seconds. However, you can use a chunking algorithm to handle longer audio files. It’s similar to cutting a longer movie into shorter scenes for easier viewing.

Here’s an example of how to implement chunking:

from transformers import pipeline
from datasets import load_dataset

device = "cuda:0" if torch.cuda.is_available() else "cpu"
pipe = pipeline("automatic-speech-recognition", model="ivrit-ai/whisper-v2-d3-e3", chunk_length_s=30, device=device)

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = ds[0]["audio"]
prediction = pipe(sample.copy(), batch_size=8)[text]

Troubleshooting

If you encounter any issues while using the Whisper ASR model, here are some common troubleshooting suggestions:

• Ensure all dependencies are installed correctly.
• Check if your audio files are in the correct format and within the desired sampling rate.
• If you receive errors related to GPU usage, ensure your system has a CUDA-compatible GPU and that PyTorch has been configured properly.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

By following these steps, you should be well-equipped to harness the power of the Whisper ASR model for your transcription and translation needs. At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions.

Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.