daanzu / deepspeech-websocket-server

Server & client for DeepSpeech using WebSockets for real-time speech recognition in separate environments

DeepSpeech WebSocket Server

[GitHub is currently matching all my donations $-for-$.]

This is a WebSocket server (& client) for Mozilla's DeepSpeech, to allow easy real-time speech recognition, using a separate client & server that can be run in different environments, either locally or remotely.

Work in progress. Developed to quickly test new models running DeepSpeech in Windows Subsystem for Linux using microphone input from host Windows. Available to save others some time.

Features

• Server
  • Tested and works with DeepSpeech v0.7 (thanks @Kai-Karren)
  • Streaming inference via DeepSpeech v0.2+
  • Streams raw audio data from client via WebSocket
  • Multi-user (only decodes one stream at a time, but can block until decoding is available)
• Client
  • Streams raw audio data from microphone to server via WebSocket
  • Voice activity detection (VAD) to ignore noise and segment microphone input into separate utterances
  • Hypnotizing spinner to indicate voice activity is detected!
  • Option to automatically save each utterance to a separate .wav file, for later testing
  • Need to pause/unpause listening? See here.

Installation

This package is developed in Python 3. Activate a virtualenv, then install the requirements for the server and/or client, depending on usage:

pip install -r requirements-server.txt
### AND/OR ###
pip install -r requirements-client.txt

To run the server in an environment, you also need to install DeepSpeech, which requires choosing either the CPU xor GPU version:

pip install deepspeech
### XOR ###
pip install deepspeech-gpu

Upgrade to the latest DeepSpeech with pip install deepspeech --upgrade (or gpu version). This package works with v0.3.0.

The client uses pyaudio and portaudio for microphone access. In my experience, this works out of the box on Windows. On Linux, you may need to install portaudio header files to compile the pyaudio package: sudo apt install portaudio19-dev . On MacOS, try installing portaudio with brew: brew install portaudio .

Server

> python server.py --model ../models/daanzu-6h-512l-0001lr-425dr/ -l -t
Initializing model...
2018-10-06 AM 05:55:16.357: __main__: INFO: <module>(): args.model: ../models/daanzu-6h-512l-0001lr-425dr/output_graph.pb
2018-10-06 AM 05:55:16.357: __main__: INFO: <module>(): args.alphabet: ../models/daanzu-6h-512l-0001lr-425dr/alphabet.txt
TensorFlow: v1.6.0-18-g5021473
DeepSpeech: v0.2.0-0-g009f9b6
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2018-10-06 05:55:16.358385: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-10-06 AM 05:55:16.395: __main__: INFO: <module>(): args.lm: ../models/daanzu-6h-512l-0001lr-425dr/lm.binary
2018-10-06 AM 05:55:16.395: __main__: INFO: <module>(): args.trie: ../models/daanzu-6h-512l-0001lr-425dr/trie
Bottle v0.12.13 server starting up (using GeventWebSocketServer())...
Listening on http://127.0.0.1:8080/
Hit Ctrl-C to quit.

2018-10-06 AM 05:55:30.194: __main__: INFO: echo(): recognized: 'alpha bravo charlie'
2018-10-06 AM 05:55:32.297: __main__: INFO: echo(): recognized: 'delta echo foxtrot'
2018-10-06 AM 05:55:54.747: __main__: INFO: echo(): dead websocket
^CKeyboardInterrupt

> python server.py -h
usage: server.py [-h] -m MODEL [-a [ALPHABET]] [-l [LM]] [-t [TRIE]] [--lw LW]
                 [--vwcw VWCW] [--bw BW] [-p PORT]

optional arguments:
  -h, --help            show this help message and exit
  -m MODEL, --model MODEL
                        Path to the model (protocol buffer binary file, or
                        directory containing all files for model)
  -a [ALPHABET], --alphabet [ALPHABET]
                        Path to the configuration file specifying the alphabet
                        used by the network. Default: alphabet.txt
  -l [LM], --lm [LM]    Path to the language model binary file. Default:
                        lm.binary
  -t [TRIE], --trie [TRIE]
                        Path to the language model trie file created with
                        native_client/generate_trie. Default: trie
  --lw LW               The alpha hyperparameter of the CTC decoder. Language
                        Model weight. Default: 1.5
  --vwcw VWCW           Valid word insertion weight. This is used to lessen
                        the word insertion penalty when the inserted word is
                        part of the vocabulary. Default: 2.25
  --bw BW               Beam width used in the CTC decoder when building
                        candidate transcriptions. Default: 1024
  -p PORT, --port PORT  Port to run server on. Default: 8080

Client

λ py client.py
Listening...
Recognized: alpha bravo charlie
Recognized: delta echo foxtrot
^C

λ py client.py -h
usage: client.py [-h] [-s SERVER] [-a AGGRESSIVENESS] [--nospinner]
                 [-w SAVEWAV]

Streams raw audio data from microphone with VAD to server via WebSocket

optional arguments:
  -h, --help            show this help message and exit
  -s SERVER, --server SERVER
                        Default: ws://localhost:8080/recognize
  -a AGGRESSIVENESS, --aggressiveness AGGRESSIVENESS
                        Set aggressiveness of VAD: an integer between 0 and 3,
                        0 being the least aggressive about filtering out non-
                        speech, 3 the most aggressive. Default: 3
  --nospinner           Disable spinner
  -w SAVEWAV, --savewav SAVEWAV
                        Save .wav files of utterences to given directory

Contributions

Pull requests welcome.

Contributors:

• @Zeddy913

deepspeech-websocket-server's Issues

Concurrent streams cause strange recognitions

Should investigate.

move audio writing to server from client

I would like the server to be responsible for logging new audio files to disk instead of the client so that it can intercept incoming requests from both the file API as well as WebSocket. Additionally, this utilizes the EOS marker that clearly indicates that the server should stop recording a stream.

• transplant code using this function into something around this point

This will be a self-completed issue, but seeking opinion before execution.

500 Internal Server Error on localhost?

Hi,

I'm getting the following error when i run the following command on anaconda prompt (Windows 10):

python server.py --model models/output_graph.pbmm --alphabet models/alphabet.txt --lm models/lm.binary --trie models/trie

...and on the localhost, when i navigate to http://127.0.0.1:8080/recognize i get the 500 error:

I have installed all requirements, including pyaudio and the latest deepspeech with pip install deepspeech and pip install deepspeech --upgrade I downloaded the pretrained model from https://github.com/mozilla/DeepSpeech/releases, tried with deepspeech-0.5.1-models.tar.gz, deepspeech-0.5.0-models.tar.gz and deepspeech-0.3.0-models.tar.gz. Still no luck.

Is this a bug? or does someone have a solution for this?

Thanks.

Can we make this code suit for deepspeech0.1?

Can we make this code suit for deepspeech0.1?

Support for keyword spotting while real time inference

Hi @daanzu Is there a possibility on how we can add keyword spotting while inferencing in real time. Like spotting custom words or giving priority to certain words on the fly just like phrases in Google Speech API. Also, would like to ask whether has Deepspeech come up with real time in the original version? Quick answers appreciated.

allowing nested calls after recognition

I would like to chain additional processing steps after the recognition has been completed. This allows the inclusion of other cool things to be executed on top of the speech alone: sentiment analysis, topic understanding, speaker detection, etc.

Here's a rough sketch of the concept...

• Each may follow their own "server" file that launches a new service so that they don't complicate the existing single-server architecture
• Each would communicate over web calls (REST) to avoid process confusion; in the future, we could expand it to be something more rigorous like a message queue.
• Each can communicate via stored JSON/metadata or audio files written to disk
• Each can "register" itself with the main speech server as a secondary process on start-up. For example, the "speaker detection" module will (a) launch it's own service, (b) register with primary server, (c) accept REST calls and reply with JSON / text as required
• Just one example, but each module could leverage other OSS like uis-rnn or pyannote-audio (both taken from this great repo of examples)

Seeking opinions at this point with more details to be flushed out later. Of course, eventually we may convert this suite into a package (e.g. satisfying #2), but that's not paramount right now.

Give client a Python interface

The sound detection and websocket streaming you've implemented is fantastic. However I'm interested in using the client in another Python application. Do you think we could look into cleaning it up and making it importable? The CLI should stay obviously, but ideally there'd be a single entry point function that could be called by some other Python code, or by the CLI/argparse interface.

I'll write a PR for this if you're interested (and once I get some time).

Update to Coqui STT

Hi there @daanzu 👋

this websocket server is an excellent project... thank you for sharing it with the open source community 💚 I used it some back when I was working on DeepSpeech:)

You might not know, but Coqui STT is the new, actively maintained fork of DeepSpeech. It would be a fairly easy upgrade, and the new Coqui v1.0 model is much more accurate than the old DeepSpeech models 😄

For more context: mozilla/DeepSpeech#3693

Feature request: ability to easily pause listening

I'll try to have an explore myself but I was wondering if there was a straightforward approach that would allow for the listening to be paused.

Would something with calling stop_stream on the Pyaudio stream and then later starting it hi again work?

Alternatively if there's progress on #2 then I could possibly integrate it that way, having the process start and stop when required.

How to speed up performance?

Hi, I would like to know,
How to speed up the performance of websocket?
Is there any parameter that we can tweak?
Does deepspeech and deepspeech-gpu make a difference?
I mean during loading the model at the server.

Thanks for reply.

can't get server running

deepspeech is running fine on my WSL.
Trying to point the server to the right directory but it won't recognize. I tried different paths but still the same. Is there anything special?

python deepspeech-websocket-server/server.py --model deepspeech-0.5.1-models/ -l -t
Initializing model...
2019-10-29 PM 04:44:28.804: main: INFO: (): ARGS.model: deepspeech-0.5.1-models/output_graph.pb
2019-10-29 PM 04:44:28.805: main: INFO: (): ARGS.alphabet: deepspeech-0.5.1-models/alphabet.txt
Traceback (most recent call last):
File "deepspeech-websocket-server/server.py", line 60, in
model = deepspeech.Model(ARGS.model, N_FEATURES, N_CONTEXT, ARGS.alphabet, BEAM_WIDTH)
AttributeError: 'module' object has no attribute 'Model'