webmachinelearning / webnn-native Goto Github PK
View Code? Open in Web Editor NEW🧠⚙️ Standalone native implementation of the Web Neural Network API
License: Apache License 2.0
🧠⚙️ Standalone native implementation of the Web Neural Network API
License: Apache License 2.0
Now we make a workaround to support fused clamp. OpenVINO can fuse clamp by its graph compiler and DML doesn't support fuse clamp today. So We added a clamp node in GraphBuilder directly to ensure
that we can find the min and max operands from the graph. We need to refactor codes once a backend requires fusing clamp.
The implementation for Op Max needs to be completed.
WebNN-native is a standalone component to implement Web Neural Network API, we have exposed the JavaScript API with Node.js Binding. We start to integrate WebNN-native into Chromium browser, there are two approaches to implement IPC between render process and gpu process
Mojom is simple but maybe the interoperability with WebGPU/WebGL is hard, The implementation of WebNN with Command buffer is complex that is designed for web Graphics.
Build debug version and run conv2d or pool2d tests:
$ out/Debug/webnn_end2end_tests --gtest_filter=*Conv2dTests*
Note: Google Test filter = *Conv2dTests*
[==========] Running 66 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 66 tests from Conv2dTests
[ RUN ] Conv2dTests.Conv2dWithPaddingDefault
webnn_end2end_tests: /home/nhu/code/webnn-native/third_party/openvino/ienn/src/ie_model.cpp:414: ie_operand_t* InferenceEngine::Model::AddConv2d(ie_operand_t*, ie_operand_t*, ie_conv2d_options_t*): Assertion `0' failed.
Aborted (core dumped)
The crash is caused by setting autoPad as explicit for conv2d and pool2d. The code of conv2d to handle explicit case should break instead of passing through to default case. Same to pool2d.
Calculate output shape of each op to meet the requirement of some backends' ( such as NNAPI) implementation.
When input rank == 3 and keeDimensions == false, reduceMean produces a result with wrong shape. This will be fixed by openvinotoolkit/openvino#5986 in OpenVINO 2021/4. Now the latest version is 2021/3, please wait for OpenVINO release version upgrading, then it will be fixed.
Based on dml backend, running npm run test-ops
through node.js under webnn-native/node
will get results as below:
test element-wise unary operations
√ abs
√ ceil
a(-0.3924400508403778) b(-0.39242464) delta(0.00001541084037781637)
59) cos
√ exp
√ floor
√ log
√ neg
a(0.9874919056892395) b(0.98748258) delta(0.000009325689239503276)
60) sin
a(6.260713577270508) b(6.26066374) delta(0.00004983727050777986)
61) tan
There are 455 JS test cases that test APIs, ops and models for webnn-polyfill. Thanks to mocha, developers can also use npm test
to run these tests by node.js, e.g. for continuous integration.
To avoid the duplicated work and porting of these tests to C++, it would be useful to run these JS tests against webnn-native. As webnn.h is a one-to-one mapping of WebNN spec, this is possible to add a node.js binding to webnn-native and expose the WebNN JS API through this binding.
According to the first wave models, there are 4 vision models including SqueezeNet, MobileNetV2, ResNetV2 and TinyYOLOV2.
To support those models, there are 5 ops need to be implemented. They are batchNormalization
, clamp
, concat
, gemm
and leakyRelu
.
These ops are supported by webnn-polyfill. It would be good to keep them aligned.
We defined tolerance as <0.005 here, but the results of resNet NCHW model tests only meet <0.01. So we need discussion on how to solve this problem.
[----------] 3 tests from ResNetNchwTests
[ RUN ] ResNetNchwTests.NchwTest0
Error: The output value at index 5 is expected as -9.77821, but got -9.77252
../../src/tests/end2end/models/ResNetNchw.cpp(32): error: Value of: utils::CheckValue(result, outputNpy.as_vec<float>())
Actual: false
Expected: true
[ FAILED ] ResNetNchwTests.NchwTest0 (3628 ms)
[ RUN ] ResNetNchwTests.NchwTest1
Error: The output value at index 26 is expected as -19.3955, but got -19.3886
../../src/tests/end2end/models/ResNetNchw.cpp(32): error: Value of: utils::CheckValue(result, outputNpy.as_vec<float>())
Actual: false
Expected: true
[ FAILED ] ResNetNchwTests.NchwTest1 (3197 ms)
[ RUN ] ResNetNchwTests.NchwTest2
Error: The output value at index 5 is expected as -8.86224, but got -8.8562
../../src/tests/end2end/models/ResNetNchw.cpp(32): error: Value of: utils::CheckValue(result, outputNpy.as_vec<float>())
Actual: false
Expected: true
[ FAILED ] ResNetNchwTests.NchwTest2 (3144 ms)
[----------] 3 tests from ResNetNchwTests (9969 ms total)
Align with latest Spec webmachinelearning/webnn#205
Now OperandBase*(or operatorBase*) in operand options struct hasn't been initialized, as below:
struct ClampOptions {
OperandBase* minValue;
OperandBase* maxValue;
};
We may need initialize them to nullptr.
Align to latest change of pool2d operations: webmachinelearning/webnn#208
It would be able to run the fast style transfer example of webnn-samples on Electron.js via node.js binding to WebNN-native.
It would be able to run the semantic segmentation example of webnn-samples on electron.js through node.js binding to WebNN-native.
Need to re-implement clamp with dml::Clip when the min and max parameters are both float values as following description:
struct DML_ELEMENT_WISE_CLIP_OPERATOR_DESC {
const DML_TENSOR_DESC *InputTensor;
const DML_TENSOR_DESC *OutputTensor;
const DML_SCALE_BIAS *ScaleBias;
FLOAT Min;
FLOAT Max;
};
Also need to refactor Clamp implementation on IE.
Implement the spec change introduced by webmachinelearning/webnn#188
The corresponding webnn-polyfill issue is webmachinelearning/webnn-polyfill#92
Using directory INTEL_OPENVINO_DIR="/opt/intel/openvino_2021"
here which links to latest 2021.4 version to build ienn will fail. INTEL_OPENVINO_DIR="/opt/intel/openvino_2021.2.185"
has been verified, so please wait for our support of 2021.4.
Reported by @Honry , thanks!
Test: https://honry.github.io/webnn-samples/semantic_segmentation/ + Electron.js + WebNN OV/DML
Relevant PR: Honry/webnn-samples#14
The root cause is related to upstream issue for building node.js addon against electron on Windows: electron/electron#29893. We rely accessing v8::SharedArrayBuffer::GetBackingStore() in fix #107. However, it turns out it doesn't work on Electron 13+ on Windows because it is built by clang/libc++. Eventually, we should use N-API to access SharedArrayBuffer, however that capability depends on nodejs/node#23276.
For short term, I would suggest to disable the SharedArrayBuffer support till it is officially supported by N-API.
$ ninja -C out/Release/
ninja: Entering directory `out/Release/'
[1/15] ACTION //src/webnn_native:build_ngraph_c_api(//build/toolchain/linux:clang_x64)
/home/nhu/code/webnn-native/third_party/openvino/ngraph_c_api/src/ngraph_c_api.cpp: In function ‘IEStatusCode ngraph_constant(const tensor_desc_t*, const ie_blob_t*, ngraph_node_t**)’:
/home/nhu/code/webnn-native/third_party/openvino/ngraph_c_api/src/ngraph_c_api.cpp:168:21: warning: ignoring return value of ‘IEStatusCode ie_blob_get_buffer(const ie_blob_t*, ie_blob_buffer_t*)’, declared with attribute warn_unused_result [-Wunused-result]
ie_blob_get_buffer(blob, &buffer);
~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
/home/nhu/code/webnn-native/third_party/openvino/ngraph_c_api/src/ngraph_c_api.cpp: In function ‘IEStatusCode ngraph_average_pool(const ngraph_node_t*, const size_t*, uint32_t, const size_t*, uint32_t, const size_t*, uint32_t, ngraph_auto_pad, ngraph_node_t**)’:
/home/nhu/code/webnn-native/third_party/openvino/ngraph_c_api/src/ngraph_c_api.cpp:457:56: warning: ‘auto_pad’ may be used uninitialized in this function [-Wmaybe-uninitialized]
true, ngraph::op::RoundingType::FLOOR, GetAutoPad(mode));
~~~~~~~~~~^~~~~~
/home/nhu/code/webnn-native/third_party/openvino/ngraph_c_api/src/ngraph_c_api.cpp: In function ‘IEStatusCode ngraph_max_pool(const ngraph_node_t*, const size_t*, uint32_t, const size_t*, uint32_t, const size_t*, uint32_t, ngraph_auto_pad, ngraph_node_t**)’:
/home/nhu/code/webnn-native/third_party/openvino/ngraph_c_api/src/ngraph_c_api.cpp:477:50: warning: ‘auto_pad’ may be used uninitialized in this function [-Wmaybe-uninitialized]
ngraph::op::RoundingType::FLOOR, GetAutoPad(mode));
~~~~~~~~~~^~~~~~
/home/nhu/code/webnn-native/third_party/openvino/ngraph_c_api/src/ngraph_c_api.cpp: In function ‘IEStatusCode ngraph_convolution(const ngraph_node_t*, const ngraph_node_t*, const size_t*, uint32_t, const int32_t*, uint32_t, const size_t*, uint32_t, ngraph_auto_pad, ngraph_node_t**)’:
/home/nhu/code/webnn-native/third_party/openvino/ngraph_c_api/src/ngraph_c_api.cpp:498:35: warning: ‘auto_pad’ may be used uninitialized in this function [-Wmaybe-uninitialized]
dilations_vector, GetAutoPad(mode));
~~~~~~~~~~^~~~~~
/home/nhu/code/webnn-native/third_party/openvino/ngraph_c_api/src/ngraph_c_api.cpp: In function ‘IEStatusCode ngraph_group_convolution(const ngraph_node_t*, const ngraph_node_t*, const size_t*, uint32_t, const int32_t*, uint32_t, const size_t*, uint32_t, ngraph_auto_pad, ngraph_node_t**)’:
/home/nhu/code/webnn-native/third_party/openvino/ngraph_c_api/src/ngraph_c_api.cpp:519:35: warning: ‘auto_pad’ may be used uninitialized in this function [-Wmaybe-uninitialized]
dilations_vector, GetAutoPad(mode));
~~~~~~~~~~^~~~~~
[6/6] STAMP obj/all.stamp
Track discussion #1 (comment) of #1 review.
The DML device wrapper code under src/webnn_native/dml/deps
is based on the Python bindings for DirectML samples. We proposed to make it more friendly for C++ usage. The DirectML is already a DEPS. So once the generic device wrapper is available in DirectML repo, we will use that and remove these files.
@BruceDai , could you please take a look of this task? It would help the node.js development.
DML doesn't support reduceMean with keepDimensions == false, it always keep dimensions. So I must manually remove the reduced dimensions in implementation. While by this way, if axes is empty which means reducing all dimensions according to reduceMean Spec, the output dimensions will be {1}. DML can not Reinterpret( just like reshape) a node for empty shape{}.
Test case: #52
When test DeepLab sample with NHWC layout and OpenVino backend (Node.js Electron app), we failed to build the graph at resample
op at following code line:
const resample0 = this.builder_.resample(
conv4, {sizes: [1, 65, 65, 256], mode: 'linear'});
While this works on DML backend, and NCHW layout also works on OpenVino backend.
Error Log:
Error: Failed to build graph.
at DeepLabV3MNV2Nhwc.build (deeplabv3_mnv2_nhwc.js:151)
at main (main.js:334)
at async window.onload (index.html?undefined:205)
Align to latest change of pool2d operations: webmachinelearning/webnn#208
Implement the spec change of webmachinelearning/webnn#123.
The changes include:
- Extend the
conv2d
operation to support transposed convolution, an essential upsample tool for encoder-decoder models.- Add
instanceNormalization
in addition to batch-normalization. Instance-normalization is a fused operation for a normalization subgraph that computes the mean and variance values per-feature instance on the fly.- Replace the
sqrt
unary operation with a more genericpow
binary operation used by the normalization process.- Add
pad
operation that supports all 4 padding modes found in various frameworks.- Add
resample
operation to support both upsampling and downsampling of feature instances. This operation is used in the ONNX version of the style-transfer models.
The corresponding webnn-polyfill issue: webmachinelearning/webnn-polyfill#34
The style transfer sample: https://webmachinelearning.github.io/webnn-samples/style_transfer/
[ FAILED ] Conv2dTests.Conv2dTransposeDefault
[ FAILED ] Conv2dTests.Conv2dTransposeNchwHwio
[ FAILED ] Conv2dTests.Conv2dTransposeNchwOhwi
[ FAILED ] Conv2dTests.Conv2dTransposeNchwIhwo
[ FAILED ] Conv2dTests.Conv2dTransposeNhwcOihw
[ FAILED ] Conv2dTests.Conv2dTransposeNhwcHwio
[ FAILED ] Conv2dTests.Conv2dTransposeNhwcOhwi
[ FAILED ] Conv2dTests.Conv2dTransposeNhwcIhwo
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputShapeDefault
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputShapeNchwHwio
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputShapeNchwOhwi
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputShapeNchwIhwo
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputShapeNhwcOihw
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputShapeNhwcHwio
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputShapeNhwcOhwi
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputShapeNhwcIhwo
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputPaddingDefault
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputPaddingNchwHwio
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputPaddingNchwOhwi
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputPaddingNchwIhwo
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputPaddingNhwcOihw
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputPaddingNhwcHwio
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputPaddingNhwcOhwi
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputPaddingNhwcIhwo
[ FAILED ] Conv2dTests.Conv2dTransposeWithAutoPadSameUpperDefault
[ FAILED ] Conv2dTests.Conv2dTransposeWithAutoPadExplicitNhwcIhwo
[ FAILED ] Conv2dTests.Conv2dTransposeWithAutoPadSameLowerNhwcIhwo
[ FAILED ] Conv2dTests.Conv2dTransposeWithOutputSizeIgnoredOutputPadding
Error message:
error: Value of: utils::CheckValue(result, expected.value)
Actual: false
Expected: true
@miaobin PTAL, thanks.
Track the discussion #1 (comment) of #1 review.
Upstream the files under src/webnn_native/openvino/ienn
to OpenVINO project. Add a DEPS and remove them once they land in the upstream repo.
The PR for Op Min based on IE has been proposed.
I verified that since this commit, I failed to run npm run test-models
under webnn-native/node
. It seems that clamp will cause memory leak.
You can reproduce by these steps:
'node/third_party/webnn-polyfill': { 'url': '{github_git}/webmachinelearning/webnn-polyfill.git@73ac6627c19b6e1f640b161166137c7b5e40698a' },
cd webnn-native/node
npm install --webnn_native_lib_path="../out/Release"
npm run build --webnn_native_lib_path="../out/Release"
npm run test-models
The use case is to allow node.js/Electron.js apps to create WebNN GPU context that is backed by DirectML and CPU context that is backed by OpenVINO.
I am trying to run webnn_end2end_tests.exe
where webnn_native component gets built under the .\third_party
sub-directory like how most components of Chromium reside (ex. third_party\Dawn
).
But when I run webnn_end2end_tests.exe
, built using webnn_enable_dml = true
edited in build_overrides/webnn_features.gni
, the following output returns:
[==========] Running 194 tests from 32 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from AddTests
[ RUN ] AddTests.AddConstantAndInput
Detected memory leaks!
Dumping objects ->
{2566} normal block at 0x000001B988199940, 12 bytes long.
Data: < > 03 00 00 00 04 00 00 00 05 00 00 00
{2564} normal block at 0x000001B98819A160, 12 bytes long.
Data: < > 03 00 00 00 04 00 00 00 05 00 00 00
...
And the test fails (does not continue to run).
Any idea how to proceed (maybe memory leak checking is causing an issue)?
webmachinelearning/webnn#149 introduces
The corresponding webnn-polyfill issue is webmachinelearning/webnn-polyfill#42
Align with latest Spec webmachinelearning/webnn#205
One case failed as below:
[ RUN ] GraphValidationTest.BuildCallBackSuccess
unknown file: Failure
Unexpected mock function call - returning directly.
Function call: Call(0, 0x558ee7392c20, NULL, 0x558ee738e6c0)
Google Mock tried the following 1 expectation, but it didn't match:
../../src/tests/unittests/validation/GraphValidationTests.cpp:65: EXPECT_CALL(*mockGraphBuildCallback, Call(MLBuildGraphStatus_Error, _, _, this))...
Expected arg #0: is equal to 1
Actual: 0
Expected: to be called once
Actual: never called - unsatisfied and active
../../src/tests/unittests/validation/GraphValidationTests.cpp:65: Failure
Actual function call count doesn't match EXPECT_CALL(*mockGraphBuildCallback, Call(MLBuildGraphStatus_Error, _, _, this))...
Expected: to be called once
Actual: never called - unsatisfied and active
[ FAILED ] GraphValidationTest.BuildCallBackSuccess (9 ms)
Node.js would throw "Invalid argument" error when passing typed array buffer backing by SharedArrayBuffer
to WebNN-native Node.js addon API, in particular, the builder.constant
and graph.compute
. The SharedArrayBuffer
is commonly used for web worker. This issue would impact the Wasm Lib, e.g. TFLite Web and ONNXRuntime Web, multi-thread build.
Following WebNN-native Binding for Node.js README, when running TinyYolov2 model tests with Node Binding of WebNN API by DirectML backend, test failed with following assertion error:
test tinyYolov2 nchw
a(0.2928530275821686) b(0.2941698133945465) delta(0.0013167858123779297)
1) test_data_set_0
a(0.4636850357055664) b(0.4621993899345398) delta(0.0014856457710266113)
2) test_data_set_1
a(0.1699068546295166) b(0.16871242225170135) delta(0.0011944323778152466)
3) test_data_set_2
test tinyYolov2 nhwc
a(0.007876233197748661) b(0.002362951636314392) delta(0.005513281561434269)
4) test_data_set_0
a(-0.017894059419631958) b(-0.011623484082520008) delta(0.00627057533711195)
5) test_data_set_1
a(-0.13853715360164642) b(-0.14467230439186096) delta(0.006135150790214539)
6) test_data_set_2
This issue isn't reproduced by Node Binding for OpenVINO backend.
Test Device info:
ASUS ZenBook Flip S laptop -- CPU:11th Gen Intel i7-1165G7, GPU:Intel Xe Graphics (driver version 30.0.100.9684)
Windows specifications:
Edition:Windows 10 Version:2004 OS build:19041.1110
These tests report unhandled NOT_IMPL errors when using DML:
[ FAILED ] PowTests.Sqrt1d
[ FAILED ] PowTests.Sqrt3d
[ FAILED ] PowTests.Pow1d
[ FAILED ] PowTests.PowBroadcastScalar
[ FAILED ] PowTests.PowBroadcast1d
We should suppress them if failure is expected so we can keep the bots green. This is particularly important for WebNN integrations.
Implement the changes introduced by webmachinelearning/webnn#174
The changes include:
- Change
MLGraphBuilder.build
andMLGraph.compute
to sync API.- Change
MLGraph.compute
to require pre-allocated output buffers.- Simplify the the input / output resources binding by
MLResource
. LeaveMLInput
only for dynamic input shape case.
The corresponding webnn-polyfill issue webmachinelearning/webnn-polyfill#81
Code at this line can't print error messages. This will cause we can't get useful error log from ValidateAndInferTypes() for each ops.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.