Comments (3)
Thanks @vedran for this relevant suggestion and your wish to contribute.
Indeed I haven't spent effort in the "verbose" option so far.
There are currently two ways of getting word timestamps:
- doing it on the fly while whisper is decoding (which is only possible with "greedy" decoding)
- letting whisper decode everything and then aligning segment by segment by running inference again (which is required when whisper do beam search, temperature fallback...)
(2 corresponds to options --accurate
and --naive
of the CLI)
With (1), it would be easy to do the verbose "on the fly" (what you call "real time").
I can do it. Just I need to decide whether or not to disable whisper verbose, i.e. to print the timestamped segment or not before timestamped words are printed.
Do you have an opinion on this?
With (2), printing "on the fly" is not possible.
The real solution would be to fork whisper and add the word alignment in it. I think I will end up with this solution in the end, for several reasons.
from whisper-timestamped.
I looked into this more carefully.
Actually it's super tricky to implement "on-the-fly" verbose for (1), because when hooking whisper transcription, we don't know the position of the current "30 sec chunk of audio" being decoded (or it's hard to recover... maybe possible).
I improved a bit things around the verbose option. Well, choosing not to print the segments returned by whisper (to avoid confusion between segments and words). Not sure it improves things for your use case...
So again, forking whisper will solve many things including this issue...
from whisper-timestamped.
Thanks for that! I will explore forking whisper as well. For my use case I want to see how fast one can push the performance of realtime processing for a transcription service. e.g. uploading a video file in chunks, passing those chunks into whisper, and returning them back to the client very quickly. For a proof of concept I just fed the whole file into your whisper version here, and parsed the whisper's verbose output, and skipped over your library's verbose output.
from whisper-timestamped.
Related Issues (20)
- everytime I update this, it bricks my python install HOT 1
- Beam Search Decoding How to Get Beam of Tokens as Output HOT 3
- Error when using -vad_v3.1 HOT 1
- Consider using whisper-distilled HOT 2
- Publication on Pypi failing HOT 7
- Is there a way to use it with whisper.cpp HOT 2
- Cannot find audio file HOT 3
- Only part of audio transcribed HOT 4
- Trouble transcribing list of files HOT 2
- torch hub path is not properly set HOT 1
- Broken link for plotting word alignment section HOT 7
- Loading finetuned model serialized with safetensors (and/or sharded models) HOT 10
- How to activate flash attention? HOT 2
- Could it be possible to apply the same technique to the whisper API? HOT 6
- ctranslate2 support HOT 1
- CPU only light install links are broken? HOT 3
- Issue with accented characters coming up as symbols in output json file
- Repetitive Phrase Looping HOT 3
- Bad timestamp prediction with some finetuned Whisper models HOT 9
- How to use cuda? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from whisper-timestamped.