ibelem / webnnmeeting

[Deprecated] Please visit https://github.com/ibelem/hybrid-work for up to date version of W3C WebNN API.

Deprecated

Please visit https://github.com/ibelem/webmlmeet for up to date version.

WebNN Meeting (Intelligent Collaboration)

A Web-based Intelligent Collaboration (Video Conference) example with AI features based on W3C Web Neural Network API and powered by Intel Open WebRTC Toolkit (OWT).

Web Neural Network (WebNN) API is a dedicated low-level API for neural network inference hardware acceleration. It is published by the W3C Machine Learning for the Web Community Group.

Web Machine Learning Features

• Blur my background
• Change my background
• More to come

Normal Video Conference Features

• Video and audio
• Screen sharing
• Conversation / Text message
• ...

Screenshot

Supported Backends for Web Machine Learning Features

• Slowest: WebAssembly (WASM)
• Slower: WebGL (Will upgrade to WebGPU later)
• Fastest: Web Neural Network (WebNN) API

Mordern browsers including Google Chrome doesn't support WebNN API now, you can try WebGL backend automatically with your browser.

Optional

If you are interested, please refer to WebNN Chromium build repo and WIKI:

• How to build WebNN Chromium on Windows, Linux
• How to run Chromium builds with WebNN API.

Precondition - Intel Open WebRTC Toolkit (OWT) Server Setup

The WebNN Meeting requires WebRTC server support, please read Server.md for setting up Intel Open WebRTC Toolkit (OWT) Server on CentOS* 7.6 or Ubuntu 18.04 LTS.

HTTPS Configuration

Go to project main dir like /home/belem/github/webnnmeeting and create private and public key, the certificate path need to be filled in certificate.key and certificate.cert of config.js.

$ openssl genrsa 2048 > webnnmeeting.key
$ chmod 400 webnnmeeting.key
$ openssl req -new -x509 -nodes -sha256 -days 365 -key webnnmeeting.key -out webnnmeeting.crt

Environment Configuration

Please update configurations in config.js under main folder.

In 'Launch the OWT Server as Single Node' section of Open WebRTC Toolkit (OWT) Server Setup, you could get values of webrtcserver.id, webrtcserver.key and webrtcserver.url like xxx.xxx.xxx.xxx or https://<webrtcserver.com>.

Run following commands,

$ npm install
$ node server/meetingserver.js

you could get the sampleRoom id like 5df9d3661b3282c0ef1a5ee3 in command line log for restapiserver.sampleroomparticipantspath.

Update config.js with correct data:

  webrtcserver: {
    id: '5df9ca6f7415937c7a91d774',
    key:
      'rGtTQokQM/OeG/9oDzK9TtFjd+OOeUmFN2dZl52mvaI4cSj1waduIJB8x21Wa9MaGqtZzV1KTWBvr7heBIgSjQjQyeBWI0RFzCTSyhFtd9jmZ994xE50Gkmb2zxkQYALef8oj8do3gT/cWfOfgq1zPooCkRtbMK1xm44Avduyj4=',
    url: 'https://xxx.xxx.xxx.xxx',
    port: '3000',
    restapiport: '3004'
  },
  restapiserver: {
    host: '127.0.0.1',
    httpport: 8082,
    httpsport: 8081,
    sampleroomparticipantspath: '/rooms/5df9d3661b3282c0ef1a5ee3/participants'
  },
  nuxtserver: {
    host: '0.0.0.0',
    httpsport: 8888
  },
  certificate: {
    cert: './webnnmeeting.crt',
    key: './webnnmeeting.key'
  }

Build Setup

# install dependencies
$ npm install

# serve with hot reload at localhost:3000
$ npm run dev

# build for production and launch server
$ npm run build
$ npm run start

# generate static project
$ npm run generate

Port Conflict

If error happens related to port conflict, you could change ports in config.js.

How to Run

If you are running the code locally, the browser will show "Your connection is not private" when accessing Intel OWT server and WebNN Meeting web pages.

• Visit https://xxx.xxx.xxx.xxx:8080/socket.io/?EIO=3&transport=polling
• Click "Advanced" button -> Click "Proceed to xxx.xxx.xxx.xxx (unsafe)"
• Visit WebNN Meeting URL set in config.js, e.g: https://127.0.0.1:8888/
• Click "Advanced" button -> Click "Proceed to 127.0.0.1 (unsafe)"

Models

If you get the WebNNMeeting source code from https://github.com/intel/webml-polyfill, please download model files from semantic_segmentation/model and put them (e.g. deeplab_mobilenetv2_257_dilated.tflite) under ./static/js/webnn/ss/model/

macOS

Once you are using macOS and it says "Your connection is not private", click somewhere on the page and then blindly type thisisunsafe which will instantly bypass the warning.

License

The License of WebNN Meeting will be Apache 2.0.

Code Hacks

There are some code hacks in Intel Open WebRTC Toolkit (OWT) and Material Design Icons, please refer to code hacks.

Error Handling

If you encounter the error ENOSPC: System limit for number of file watchers reached, watch when build the project on Ubuntu, please try:

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

webnnmeeting's Issues

Failed to Initialize WebNnRoom -- ETIMEDOUT...

Failed to intialize webnnroom Error: connect ETIMEDOUT 10.239.47.52:3000
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1126:14) {
errno: 'ETIMEDOUT',
code: 'ETIMEDOUT',
syscall: 'connect',
address: '10.239.47.52',
port: 3000
}

Windows 10 home OS, version 1909
Intel i7-3770, 12.0GB ram, 64 bit o/s
$ npm --version // 6.10.3
$ node --version // v12.10.0
I'm using github bash CLI client terminal...

I will say when I first ran npm install there were a whole lot of vulnerability warnings. npm update and npm audit fix corrected most of them, rerun npm run dev = same error as above.

Am i doing this correctly? I was hoping for a local host test of the system.
many thanks...

edit, info from: webnnmeeting/config-windows.js

webrtcserver: {
    id: '5df9ca6f7415937c7a91d774',
    key:
      'rGtTQokQM/OeG/9oDzK9TtFjd+OOeUmFN2dZl52mvaI4cSj1waduIJB8x21Wa9MaGqtZzV1KTWBvr7heBIgSjQjQyeBWI0RFzCTSyhFtd9jmZ994xE50Gkmb2zxkQYALef8oj8do3gT/cWfOfgq1zPooCkRtbMK1xm44Avduyj4=',
    url: 'https://10.239.47.52',
    port: '3000',
    restapiport: '3004'
  },

WebRTC server? Am I missing something from the readme?

Server Error

I am Getting This Error While Stating the Server

root@jisu-Conv:/home/jisu_admin/webnnmeeting# npm run start

> [email protected] start /home/jisu_admin/webnnmeeting
> concurrently "nodemon server/meetingserver.js" "cross-env NODE_ENV=production node server/index.js"

[0] [nodemon] 2.0.4
[0] [nodemon] to restart at any time, enter `rs`
[0] [nodemon] watching path(s): *.*
[0] [nodemon] watching extensions: js,mjs,json
[0] [nodemon] starting `node server/meetingserver.js`
[0] sampleRoom Id: 5f7f142cb6e9f12ecafb1afa
[0] WebNN Meeting Rest API Server HTTPS Port: 8081
[1] WebNN Meeting App HTTPS Port: 8888
[1]
[1]  ERROR  Renderer resources are not loaded! Please check possible console errors and ensure dist (/home/jisu_admin/we                            bnnmeeting/.nuxt/dist/server) exists.
[1]
[1]   at VueRenderer.renderRoute (node_modules/@nuxt/vue-renderer/dist/vue-renderer.js:4400:17)
[1]   at VueRenderer.renderRoute (node_modules/@nuxt/vue-renderer/dist/vue-renderer.js:4388:21)
[1]   at async nuxtMiddleware (node_modules/@nuxt/server/dist/server.js:156:20)
[1]
 ERROR  Renderer resources are not loaded! Please check possible console errors and ensure dist (/home/jisu_admin/we                            bnnmeeting/.nuxt/dist/server) exists.

[Feature blocker] Attention: Unable to fetch remote streams caused by OWT server issue

When upgrading the Google Chrome browser to 85 or higher, you may unable to get video streams from other attendees in WebNNMeeting.

This is caused by the OWT server issue which has been fixed by setting x509 version in DTLS in https://github.com/open-webrtc-toolkit/owt-server

The new release of the CS_WebRTC_Conference_Server_MCU.v_Version_.Ubuntu.tgz with this bug fix has not been released to the public or end users, stay tuned. Will update Server.md when new public releases are available.

Need Full Documentation from the Scratch

i have Installed The WEBNN in a VPS But It is not Working
And I am Unable to find the Config.js File
Where to find The Config. JS ?

FPS drops when integrate refactored BaseRunner.js

Full Screen	Before Refactor	Before Refactor	After Refactor	After Refactor
	Inference Time(ms)	FPS	Inference Time(ms)	FPS
DeepLab 224	52.72	16	48.18	12
DeepLab 257 Atrous	45.92	17	41.75	12

Also reproduced with WebML SS Examples.