Question about CLAP score evaluation about auffusion

[happylittlecat2333]

Actually, all open-sourced CLAP are trained on 48K sample rate audio. We follow huggingface CLAP scripts to get the text_embeds and audio_embeds, and follow the original LAION-AI CLAP scripts to use librosa to upsample the audio. Therefore the code can be:

model = ClapModel.from_pretrained("laion/clap-htsat-unfused").to(device)
processor = AutoProcessor.from_pretrained("laion/clap-htsat-unfused")

audio_sample, sampling_rate = librosa.load(audio_path, sr=48000)
audio_sample = librosa.util.normalize(audio_sample)

inputs = processor(text=text, audios=audio_sample, sampling_rate=sampling_rate, return_tensors="pt", padding=True).to(device)

outputs = model(**inputs)

text_embeds = outputs.text_embeds
audio_embeds = outputs.audio_embeds

[IFICL]

Thanks for the answer and code.