isabella232 / bumblebee

Jaxcore Bumblebee - a JavaScript voice application framework

JavaScript Voice Application Framework

Write your own voice apps and assistants with an easy-to-learn JavaScript API!

About

Bumblebee is a set of libraries, tools, and methodologies that enables JavaScript developers to write their own conversational voice assistants in either NodeJS or on the web.

The open source technologies that Bumblebee uses are:

• NodeJS - V8 JavaScript engine
• ElectronJS - JavaScript desktop application framework
• Mozilla DeepSpeech - Tensorflow-based speech-to-text processing
• Picovoice Porcupine - JavaScript hotword detection
• meSpeak - JavaScript text-to-speech library

Bumblebee builds upon these technologies by making them easier to use and tying them together into an integrated voice application API. Because these systems run locally without any cloud services, stand-alone privacy-focused always-on voice applications can finally be realized.

The Bumblebee project includes an ElectronJS voice app server console, which automatically installs and sets up DeepSpeech, and runs the bumblebee websocket service that voice applications can connect to. The applications run independently of the bumblebee server, and use a websocket API to connect and communicate by receiving speech-to-text results and hotword commands, or issuing text-to-speech instructions.

There are limitless ways to expand Bumblebee's capabilities by writing new applications that control devices or services on a home network, retrieve data from the internet, and anything else you can think of. And as you will see, they are both EASIER and MORE FUN to write than you think.

Bumblebee voice apps are small, simple, single file scripts. They can be shared easily between systems and build upon eachother to create larger and smarter voice applications. By associating a voice app with a hotword, your app becomes a voice assistant that can be called upon at any time.

Installation

Requirements

The computing resources required to run Bumblebee are much larger than a typical application.

• Disk Space: 1.9 GB of space is required for installation
  • Bumblebee: ~450 MB
  • DeepSpeech: ~1.4 GB
• CPU's with AVX support are required, GPU may be utilized if available
• RAM usage will fluctuate ~350 MB (or higher)

Desktop Application

To install and run the Bumblebee desktop app from the source files, see:

• INSTALL.md

After installation, the first time Bumblebee is run it will prompt to download the DeepSpeech pre-trained English model files (1.4 GB disk space required).

When installed successfully and with the microphone and speakers turned on, the console will react in real-time to the audio it hears. Test the that speech-to-text is operational before proceeding.

Hello World

To get started, create the most simple Bumblebee app possible, a "Hello World" voice application.

Create a new directory and NPM project:

mkdir helloworld
cd helloworld
npm init
npm install jaxcore-bumblebee

Create a new file named helloworld.js:

const Bumblebee = require('jaxcore-bumblebee');

class HelloWorldApp extends Bumblebee.Application {
	constructor() {
		super(...arguments);
	}

	async loop() {
		this.console('Say "Hello World"');

		let recognition = await this.recognize();
		this.console(recognition);

		if (recognition.text === 'hello world') {
			await this.playSound('okay');
			await this.say('Hello World');
		}
		else {
			await this.playSound('error');
		}
	}
}

Bumblebee.connectApplication(HelloWorldApp, {
	name: "Hello World",
	autoStart: true
});

Run the voice app:

node helloworld.js

With the speakers and microphone turned on, you will be able to talk to this application and listen to it's responses. It doesn't do very much yet, if it hears you say "hello world' it will respond by making a beep sound and also saying "hello world".

This program will run continuously until the NodeJS script is closed using Command+C or Control+C and it can be started and stopped at any time.

Documentation (Coming Soon)

This is a brand new project with much more to come.

There are some additional examples to try and many more are in the works.

Use github's "watch" feature to stay tuned for updates!