Comments (7)
I share your observations, especially on multiple words sharing the same timestamp, which occurs very frequently in my use cases. I have set suppress_silence = False
and suppress_middle = False
to get more reliable results, but of course that just creates other inaccuracies. I think your proposed solution could certainly help to stabilize the results and avoid tight word clusters.
from stable-ts.
You raised interesting points. On top of my head, I have three solutions to address some of the issues brought up.
Let's consider the two examples you brought up by combining them and making it less favorable for the current suppression logic by making 3rd candidate slightly early.
1 2 3 4 5 6 7
| | | | | | |
######## ### ###### #############
1. Average pooling the timestamp probability distribution.
Then it becomes something like this where the posts with number on top has the higher probabilities:
1 2 3 4 5 6 7
||||| ||| |||||| ||| |||
######## ### ###### #############
2. Min pooling the suppression mask.
This way it only suppresses slightly up to before the start of a sound/speech. Thus it leaves room for timestamps right at the edges to be chosen instead of just ignored (without min pooling the small gaps at 5,6,7 would have been suppressed).
3 4 5 6 7
| |||||| ||| |||
######## ### ###### #############
3. Suppressing non-gaps and min pooling the suppression mask for that as well.
3 5 6 7
| ||| ||| |||
######## ### ###### #############
The last method will not work as well for audio that has more than just speech. If the 4th candidate is actually a gap in speech but there is background noise or some other loud sound the occur during the gap then applying this step actually makes it perform worse. I would have the disabled for word-level timestamps
from stable-ts.
Regarding background noise, note that if I used this I would definitely be replacing the suppression mask with actual VAD like silero-vad. I assumed the reason you chose not to use this was that you were just showing a proof-of-concept, but with the real intention to use proper VAD, and making it easy for us to swap out that code for real VAD should we choose to.
Regarding your ideas, I don't exactly understand how to interpret your average pooling and min pooling. Just to be precise here, what are the actual pools/batches that you take the average or min of? In the min pooling case, it sounds like you would have a left-leaning bias, in that it would only help to find the start timestamps. I would like to see some effort to also detect the end timestamps by using the suppression mask or proper VAD. In my approach, each gap (by which I mean a silent, or non-speech region), has TWO boundaries, a left and a right one. The left boundary of a gap is an attractor for the end timestamp of the preceding word, while the right boundary of a gap is an attractor for the start timestamp of the next word. It is sort of like that feature in diagramming tools where you can "snap to grid", where here we are snapping to both the left and right boundaries.
As for "Suppressing non-gaps", I also am not clear how to interpret "non-gap". If what you mean is that you want to suppress background noise that is not speech, then it is the same point as using a proper VAD.
As for average pooling the timestamp probability distribution, I am not really sure how this would work out in practice. It could be that "snap to vad" may still improve accuracy further on top of that.
Finally, regarding the problem where multiple words are merged together especially across sentence boundaries, I also want to suggest that in this situation the Whisper timestamps should probably not be respected at all and can be thrown out. The timestamps could be completely wrong, and yet it would still be safe for us to assume that a "." or a "。" should snap to the left boundary of a gap in the VAD, and the next word should start on the right boundary of that gap.
P.S. An afterthought here. I wonder if these more accurate timestamps are actually fed into the prompt for the next segment to actually help the Whisper model get more on track for the next segment. I don't think it does that.
from stable-ts.
Another thought.
Consider the following audio file with two sentences, again #
indicates a probable speech signal inferred by VAD or other means, and |
indicate candidate timestamps for the start of the second sentence.
1 2 34 5
| | || |
######## # # ################
A B C D
I am bob Nice to meet you
B and C are false positives in the speech detection. A and D are the correct designations for the first and second sentences.
The following shows how stable-ts can get this wrong:
1
|
######## ################
A B
I am bob Nice to meet you
stable-ts inference:
########|###########################################
I am bob N i c e t o m e e t y o u
For some reason, all the words in the second sentence are spread out. The same thing that occurs in Whisper when no measures are taken to mask the large silence that may exist before the first word in a sentence.
Now it is obvious to us that right after candidate timestamp 1, there is mostly silence for a very long time making it actually a poor candidate. The second sentence has enough speech in it that it would need a sizable amount of speech signal to properly be designated to that region. If we just look at the start timestamp alone, though, we can't see that.
We can estimate the average token duration by taking the total duration of all speech signals and dividing that by the total number of tokens in the audio file as inferred by Whisper. From that we can estimate the expected duration of sentence 2. And using that, we can figure out that candidate 5 is the only start timestamp that has a sufficient amount of voice signal after it to be able to fit the number of inferred tokens for that sentence.
from stable-ts.
_
is where the timestamp tokens will be suppressed. #
is non-silence. The silence suppression work like this right now:
####### ##### #### ### ####
______________________ ____ _ _ _____
After min pooling it will look something like this:
####### ##### #### ### ####
_____________________ __ ___
I had considered using VAD in place of the silence mask but adding another network to the pipeline adds more complexity, and the failures of both model will stack. You can never be certain that VAD gives you correct results but you can always determine with certainty whether a part is silent or not.
from stable-ts.
Thanks for the example, the end result looks similar to simple padding (which also has the downside I brought up earlier) but I still don't understand the calculation because I don't understand what are precisely the batches/pools that are being min'ed.
But that still doesn't leverage much of the valuable timestamp data that's available in the vad or silence mask that is described above.
Regarding vad, the problem is that most of the audio I deal with has a longish intro with music and it never works correctly. There can also be different sections of the whole audio that each have their own intro music. Silero-vad is quite reliable and overall improves the accuracy which is the main metric. Either way, the timestamps will still be off sometimes of course, but with Silero-vad, it is comparatively far more accurate, so it's an improvement that doesn't lose any parts of the transcript but just results in more accurate timestamps overall.
from stable-ts.
To deal with music, you can use a music source separation model to preprocess the audio (e.g. https://github.com/facebookresearch/demucs)
from stable-ts.
Related Issues (20)
- output both txt and subs at the same time? HOT 2
- Always a bit delay and a bit early stops HOT 4
- Faster Whisper Alignment Stuck at 0% Progress Intermittingly HOT 10
- Make Whisper dependency optional? HOT 4
- Whisper is skipping quotes HOT 4
- Issue with the timestamps. HOT 2
- IndexError while using `align()`
- Installation instructions incomplete
- Any options to prevent "flickering" HOT 3
- Attribute error in alignment
- SRT should start from 1
- Allow for padding subtitle start / stop HOT 1
- Error when using --tag with vtt format (CLI) HOT 1
- Is there anyway to make json file from the start time we want? HOT 2
- Fake Example? HOT 2
- How do I get the "string" from result? HOT 2
- I want to set threshold producing srt in order not to let the string length too long.
- Could not load library libcudnn_ops_infer.so.8. Error: libcudnn_ops_infer.so.8: cannot open shared object file: No such file or directory HOT 1
- `model.align` does not work with huggingface transformer HOT 1
- How to achieve real-time highlighted words in the demo video? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stable-ts.