Working example on Google Colab? about bark.cpp

[hswlab]

@PABannier thank you, I tried to quantize the weights and could successfully generate an output.wav
Here's the whole script for GooGle Colab, in case anyone else wants to try it out.

# install cmake
!apt update
!apt install -y cmake

# Clone bark c++ from github
%cd /content
!git clone https://github.com/PABannier/bark.cpp.git

# Build
%cd /content/bark.cpp
!mkdir build
%cd ./build
!cmake ..
!cmake --build . --config Release

# install Python dependencies
%cd /content/bark.cpp
!python3 -m pip install -r requirements.txt

# obtain the original bark and encodec weights and place them in ./models
!python3 download_weights.py --download-dir ./models

# convert the model to ggml format
!python3 convert.py \
        --dir-model ./models \
        --codec-path ./models \
        --vocab-path ./ggml_weights/ \
        --out-dir ./ggml_weights/

# Quantize weights
!mkdir ggml_weights_q4
!cp /content/bark.cpp/ggml_weights/ggml_vocab.bin /content/bark.cpp/ggml_weights_q4
!cp /content/bark.cpp/ggml_weights/ggml_weights_codec.bin /content/bark.cpp/ggml_weights_q4
!cp /content/bark.cpp/ggml_weights/vocab.txt /content/bark.cpp/ggml_weights_q4

!/content/bark.cpp/build/bin/quantize /content/bark.cpp/ggml_weights/ggml_weights_text.bin /content/bark.cpp/ggml_weights_q4/ggml_weights_text.bin q4_0
!/content/bark.cpp/build/bin/quantize /content/bark.cpp/ggml_weights/ggml_weights_coarse.bin /content/bark.cpp/ggml_weights_q4/ggml_weights_coarse.bin q4_0
!/content/bark.cpp/build/bin/quantize /content/bark.cpp/ggml_weights/ggml_weights_fine.bin /content/bark.cpp/ggml_weights_q4/ggml_weights_fine.bin q4_0

# run the inference
!/content/bark.cpp/build/bin/main -m ./ggml_weights_q4 -p "Hi this is a test with quantized weights"

# ZIP Files
!zip -r bark.zip /content/bark.cpp/build/bin /content/bark.cpp/ggml_weights /content/bark.cpp/ggml_weights_q4

# Move files to Google Drive for download
from google.colab import drive 
drive.mount('/content/drive') 
!cp /content/bark.cpp/bark.zip '/content/drive/My Drive/'

[PABannier]

Yes! Codec weights are not currently quantized because it does not provide any significant speed-ups (the forward pass is already fast), but degrades the audio quality.
I'm currently investigating other problems, so your input on this memory problem is very much welcome ;)

[AGogikar]

I tested this, the execution breaks -
bark_forward_coarse_encoder: mem per token = 8.51 MB
bark_forward_coarse_encoder: sample time = 11.67 ms
bark_forward_coarse_encoder: predict time = 197814.42 ms / 610.54 ms per token
bark_forward_coarse_encoder: total time = 198112.20 ms

[PABannier]

Hi @hswlab !
Thanks for trying on Google Colab.
I tested the prompt "this is an audio" on multiple machines and it works for me. Maybe, the CPU provided by Google Colab is not powerful enough even to run bark.cpp. A few suggestions:

• Try a different seed by passing with --seed
• Quantize the weights

[qnixsynapse]

Hi. @PABannier Sorry for asking here since I don't know how to contact you, but I want to know why bark_forward_fine_encoder tries to allocate 30GB of memory? None of the weights are at that size. Also using the original bark model(non ggml), it runs well.

fine_gpt_eval: failed to allocate 32031611289 bytes
bark_forward_fine_encoder: ggml_aligned_malloc: insufficient memory (attempted to allocate 30547.73 MB)
GGML_ASSERT: ggml.c:4408: ctx->mem_buffer != NULL

[PABannier]

Hi @akarshanbiswas ! Thanks for reaching out on this problem. Are you able to track back which operations cause this surge in memory? Also, which prompt did you input to the model?

[qnixsynapse]

I followed the same instructions the OP has in his colab notebook. Additionally I tested with quantize weights using the scripts available in the repo.

I also found out that the weight with codec in its name is not being quantized and the program crashes in the process.

I am yet to check the coredumps that I got which I will do in a few hours for now. (currently afk). 🙂

[qnixsynapse]

I moved this discussion to a separate issue.

[khimaros]

it would be great to have #95 (comment) linked or incorporated into the README

[PABannier]

@khimaros I added it to the README. Thank you @hswlab !