Node integration about cts

[kainino0x]

Yes, because I deleted that support. However it should be nearly trivial to reintroduce that support in implementation.ts, and this is the necessary path forward.

Curious, do you have a reference for the index.node addon you reference in the experimental source tree? I am using Felix's NAPI addon in it's place and am wondering how you got CTS to work with your addon.

Excellent question. Here's the abandoned patch where the code lives that created that addon.
https://dawn-review.googlesource.com/c/dawn/+/3640
It only had a few entry points.
I don't remember how this stuff works. I hope this is a helpful pointer.

[kainino0x]

I'm sorry to have implied the work is trivial. Obviously it is not. I am saying only that the amount of code that was removed from the CTS is nearly trivial and can be easily re-added by "copying" from the old code to the new code. Actually making the integration work (getting the functions to match, the API to match, etc.) is not easy.

[kainino0x]

And, of course, let me know if you run into any more issues (or if any of the stuff I just did isn't working - I didn't test it exhaustively).

[dcerisano]

Thanks @kainino0x, no problem.
I have a feeling I will be testing it exhaustively anyway.

[kainino0x]

Hey Felix, unfortunately this can't be a priority for me so it'll be at least 2 weeks before I can consider it. I think it will be pretty quick for you to do it - and easier, since you already have builds.

All you need to do is get the getGPU() function to return an object that implements requestAdapter:
https://github.com/gpuweb/cts/blob/master/src/framework/gpu/implementation.ts

I used to have support in there for my proof-of-concept node bindings (that had like 5 entry points implemented). You can look at how it was done then as an example:
https://github.com/gpuweb/cts/blob/26b96dac48eeaee39ccb62c905b34f59cfd74129/src/framework/gpu/implementation.ts

[maierfelix]

Thanks for the useful information, I will try

[dcerisano]

Hey guys, I am working on integrating Kai's CTS and Felix's node-webgpu while Felix is completing his super awesome node-webgpu RT demos.

I see that the official webgpu spec is imported into CTS from the @webgpu package. Seems that @webgpu is currently designed mostly for browser support and so does not provide for the stand-alone (eg. addon) requirement to explicitly set a window, since browsers provide one implicitly.

Felix extended the addon API by providing a WebGPUWindow. He also changed the addon GPU.requestAdapter API by passing a WebGPUWindow there instead of GPURequestAdapterOptions as specified by @webgpu.

So I think we could add GPUWindow to the @webgpu spec and also add GPUWindow in GPURequestAdapterOptions so we can keep the same signature. That would be one way of satisfying the requirement for both browser and stand-alone modes in the official @webgpu API.

I have seen on other APIs the ability to set a Graphics Context (GC), which is what a GPUWindow is. Maybe call it GPUContext instead of GPUWindow? It would be kind of neat if it could just be a Canvas or other lower level GC, allowing for WebGPU powered desktops, overlays, screensavers, widgets, wallpapers, etc.

My main focus is GPU compute, which of course does not require a GC at all. So I guess maybe @webgpu API could allow for no GC (headless) mode too?

My test branch where I am working on this is here, a clone of Kai's experimental tree he gave above. Please note I am currently focused on the linux platform and will try to keep any PRs I submit cross-platform. The changes currently in there are certainly not as 'typescripty' as they should be.

Pardon the delay getting going with this, Typescript was a surprise to me, and slowly beginning to appreciate its value, especially in a complex API like @webgpu. It already caught this issue, and kind of led to a solution.

[kainino0x]

What package are you talking about by @webgpu? Do you mean @webgpu/types? The WebGPU spec is not imported into the CTS - there is no JS code in it.

I don't see how spec changes could make the same code work in both. You'll still need to do, essentially, const gpu = (typeof navigator !== 'undefined') ? navigator.gpu : require('webgpu').

Re: GPUWindow. I don't understand what a GPUWindow is. Is it an OS window-system window? We do not want a window or canvas to create a WebGPU device. WebGPU devices should always be created headless - so they can be used with zero, one, or more canvas/window outputs, via GPUSwapChain. I think this should be possible in native for the modern APIs (though OpenGL needs a window to be initialized...).

The native webgpu.h (used for wgpu-native, Dawn, and Emscripten) defines the concept of a "surface", which can be used to create a swap chain. GPUSurface does not exist in the JS API, but it is closely related to GPUCanvasContext. webgpu-node SHOULD use the GPUSurface API to hook windows into Dawn, however, we have not finished implementing it yet.

For creating windows, webgpu-node should provide a special extra entry point (since you cannot create <canvas> elements), e.g. gpu.createWindow, which creates an OS window and returns an object that implements the GPUCanvasContext interface (configureSwapChain and getSwapChainPreferredFormat).

---

P.S. Please be careful not to @ the github user "@webgpu" - that's a project unrelated to W3C WebGPU.

[dcerisano]

@webgpu/types contains the Typescript definition file (index.d.ts) where the official (@) webgpu API is defined. Seems to be the whole point of using Typescript in CTS.

I am seeking to integrate CTS with NAPI, not a browser. CTS seems to have been created for browser validation only, and may require changes to validate non browser applications.

There still needs to be a way to specify a graphics context for non browser applications.

I will look at implementing GPUSurface, thanks - I think that is what I was looking for. Again pardon my unfamiliarity with the webgpu API which I appreciate is still a WIP. I will raise the issue in group.

Node-webgpu already provides gpu.WebGPUWindow but that is not part of the spec and there is not any way to pass it to the API. Typescript will not permit this without changing @webgpu/types.

Please note that node-webgpu does not use Typescript (does not force compliance with @webgpu/types), but maybe it should.

[kainino0x]

@webgpu/types contains the Typescript definition file (index.d.ts) where the official (@) webgpu API is defined. Seems to be the whole point of using Typescript in CTS.

That's largely the point, but also for internal type checking.

I am seeking to integrate CTS with NAPI, not a browser. CTS seems to have been created for browser validation only, and may require changes to validate non browser applications.

It was created with Node in mind, but it may have broken very slightly since then. We do compile and run the unit tests in Node on CI.

I will look at implementing GPUSurface, thanks - I think that is what I was looking for. Again pardon my unfamiliarity with the webgpu API which I appreciate is still a WIP. I will raise the issue in group.

I don't think there is any issue that needs to be raised in the group. The gpuweb group is not adding a GPUSurface concept. If you have any webgpu.h related issues they should go here: https://github.com/webgpu-native/webgpu-headers

Node-webgpu already provides gpu.WebGPUWindow but that is not part of the spec and there is not any way to pass it to the API. Typescript will not permit this without changing @webgpu/types.

You should provide a custom type for gpu and gpu.WebGPUWindow but use the @webgpu/types type definition for GPUAdapter for the return type of gpu.requestAdapter even if your requestAdapter has a different signature. That way you can customize just the part you need and use the upstream types for everything else.

---

Sorry, missed a response before:

My test branch where I am working on this is here, a clone of Kai's experimental tree he gave above. Please note I am currently focused on the linux platform and will try to keep any PRs I submit cross-platform. The changes currently in there are certainly not as 'typescripty' as they should be.

That's not an experimental branch, it's a very old revision. Please update to the latest version of the CTS but use that old code as inspiration.

[dcerisano]

That's largely the point, but also for internal type checking.

Type checking is the whole point of Typescript. There is no other point to it, and even that point is debatable. In this application, compliance with the official @webgpu API spec, which is is only defined in Typescript, AFAIK. There is also a webgpu IDL (dawn.json) but not sure if this is considered official.

It was created with Node in mind, but it may have broken very slightly since then. We do compile and run the unit tests in Node on CI.

I am integrating against NAPI, not necessarily Node. My ultimate goal is to integrate Felix's NAPI webgpu library directly with V8, not Node ("because security reasons").

You should provide a custom type for gpu and gpu.WebGPUWindow but use the @webgpu/types type definition for GPUAdapter for the return type of gpu.requestAdapter even if your requestAdapter has a different signature. That way you can customize just the part you need and use the upstream types for everything else.

It seems (again pardon my relative newness) that webgpu API can be used without upstream modification in-browser only. There does seem to be a requirement to include any web-enabled application (eg. Node, direct V8, etc). I will raise it in group, but I appreciate most (all?) of the stakeholders are browsers and this is not a priority.

That's not an experimental branch, it's a very old revision. Please update to the latest version of the CTS but use that old code as inspiration.

The source tree you provided above (which my test-branch is cloned from) is headlined as "# cts-experiment" in it's README. Also, it seems to have been updated only a few months ago, which is not "very old" or even "old" by any stretch. "Stale" might be more appropriate, but it was the only reference you gave in this thread.

At this point, I am just trying to get a single CTS unit test to run with NAPI before even thinking about any PRs.

[kainino0x]

Type checking is the whole point of Typescript. There is no other point to it. In this application, compliance with the official @webgpu API spec, which is is only defined in Typescript, AFAIK.

I'm saying that it's to check two sets of types: the @webgpu/types types for the API, and all of the types inside the CTS itself. The CTS has a considerable amount of framework code and the types massively help write correct framework code and test instantiations.

I am integrating against NAPI, not necessarily Node. My ultimate goal is to integrate Felix's NAPI webgpu library directly with V8, not Node ("because security reasons").

Good point, thanks for clarifying. That said, I don't think it will be any more difficult with V8 than with Node.

It seems (again pardon my relative newness) that webgpu API can be used without upstream modification in-browser only. There does seem to be a requirement to include any web-enabled application (eg. Node, direct V8, etc). I will raise it in group, but I appreciate most (all?) of the stakeholders are browsers and this is not a priority.

Apps that use webgpu-napi can use a .d.ts file that's packaged along with webgpu-napi, and that .d.ts can have a dependency on the upstream @webgpu/types .d.ts file.

Also note that @webgpu/types is not really maintained by the WebGPU group, it's maintained by me.

The source tree you provided above (which my test-branch is cloned from) is headlined as "# cts-experiment" in it's README.

That's because, at that time, we had not yet made the CTS "official". It was my "experiment" in creating the very first CTS for WebGPU.

Also, it seems to have been updated only a few months ago, which is not "very old" or even "old" by any stretch. "Stale" might be more appropriate, but it was the only reference you gave in this thread.

It is 7.5 months old, and a lot has changed since then. There were a miniscule number of tests at that time - almost all of the tests were added since then. The API surface of WebGPU itself has changed considerably - if webgpu-napi matches the latest webgpu.dev, it's not going to be compatible with that version of the CTS.

I didn't reference "master of the repository this issue is on" because it didn't seem necessary.

[dcerisano]

I didn't reference "master of the repository this issue is on" because it didn't seem necessary.

It was necessary for you to reference the experimental source tree because the master branch does not provide the framework in implementation.ts for testing against an out-of-browser application. Only the experimental branch attempts this.

Curious, do you have a reference for the index.node addon you reference in the experimental source tree? I am using Felix's NAPI addon in it's place and am wondering how you got CTS to work with your addon.

[dcerisano]

Hey @maierfelix ! Kai said "trivial"! LOL. Felix nearly bit my head off when I said that.

Certainly it will not be much code in the end (because I am determined to use Felix's NAPI addon - that which is actually being tested), but the complexity of the WIPs involved is staggering (even @kainino0x does not remember "how this stuff works" and he wrote CTS) on top of that the thing has to be written in Typescript.

[dcerisano]

I have done a preliminary integration of Felix's WebGPU NodeJS addon and the CTS head.

I am getting these errors when I run tools/run cts

The problem seems to be that CTS has overridden nodejs filename resolution for internal require() statements. This generates a special listing to work around browser file system limitations.

The override will not handle addons (index.node) files, which are loaded via require() statements in my implementation. I am not running within a browser, so how do I turn off the listing generator and/or nodejs override?

Thanks @kainino0x, I know you are busy and I am trying to keep my questions as focused as possible. Note that this override was not in the previous cts-experimental source tree I was using.

The addon loaded perfectly there, but the new CTS head seems to be much more browser oriented. I hope the non-browser implementations ( NodeJS addon, V8 addon, etc) of WebGPU can be tested without too much further effort.

Also would like some input on how to include CTS in Felix's NodeJS addon project and upcoming V8 addon. Or did you envision including the addons in CTS? (like I am currently doing). Felix and I thought the former would be best.

[kainino0x]

Sorry I haven't had a chance to look into this today. Sorry about the weird problem with overriding require. It is not for the browser, it is actually node specific. It allows us to write the command line tools in TypeScript and share code with the rest of the codebase.

Is the issue that it wants all files to end in .js? Maybe you can just also allow .node here: https://github.com/gpuweb/cts/blob/master/src/tools/setup-ts-in-node.js#L16
Otherwise let me know and I will take a closer look.

On including between the cts and the implementation, it would make most sense to include the cts from the implementation. But I will have to think about how to do. I don't think we'll publish the cts as a npm package, but maybe it would work to call into it as a submodule.

[dcerisano]

Completed the draft integration of CTS and NAPI WebGPU addon here.

Thanks for your hints @kainino0x, they were enough to get things going in the right direction.

Hitting a new (and better) issue now.
When I tools/run cts my desktop freezes for a minute while 790 windows are created (one for each CTS test) and then a core dump:

Warning: terminator_CreateDevice: Failed in ICD libGLX_nvidia.so.0 vkCreateDevicecall
Warning: vkCreateDevice:  Failed to create device chain.
Info: vkCreateDevice failed with VK_ERROR_INITIALIZATION_FAILED
Segmentation fault (core dumped)

I am assuming this is because I am simply running out of memory (too many windows). Again pardon my newness to CTS (ie Typescript and Babel), but is there a way to easily run the tests sequentially rather than in parallel? Or maybe re-use the window in a thread-safe manner? Or destroy each window before the next is created?

Further assuming this is not an issue with in-browser testing (does not create 790 windows).

Once we get the actual test results we can look at how such integrations should be packaged (NAPI, direct V8, etc with CTS on the various target OSes).

NOTE: I am developing this on Linux, so building Chromium+WebGPU (12 hours?!) is out of scope for me. We will need to make sure this solution still works for Chromium (pretty sure it will, the modifications while non-trivial are mercifully few).

[kainino0x]

Hm, sorry about this. I think I still have a TODO to not run all the tests at once. If that's still there I can fix it soon.
That said, we also definitely need to get those windows to close (or at least create fewer of them). We need to make sure there's a way to close the windows reliably without relying on the garbage collector (so the napi plugin will need to expose a method for that).

Does it currently create one window for every device? I currently create one device for every test case and I have another TODO to not do that.

No worries about any breaking changes for the Web version. I'm happy to help fix anything up as needed when we're merging any code changes you have.

[dcerisano]

Still poking away at this. I got one test to pass.

dcerisano@kronos:~/git/cts$ tools/run cts:canvas

** Summary **
Passed  w/o warnings = 1 / 1 = 100.00%
Passed with warnings = 0 / 1 =   0.00%
Failed               = 0 / 1 =   0.00%

It should not have passed, since it is browser based, and I am testing a NAPI addon, not in browser.

@kainino0x could you create a new branch cts-napi with those TODOs you mentioned? I still can't run any suite that has more than one test. I would like to submit PRs against that branch for now, not the master.

[kainino0x]

Sorry for the delay. I'm looking at putting some fixes in now.

[kainino0x]

It should not have passed, since it is browser based, and I am testing a NAPI addon, not in browser.

It is actually supposed to pass. It checks for the existence of document, in order to figure out whether it's running in a browser main thread. If document doesn't exist, then it skips itself (which is a "pass"). This is so that it doesn't meaninglessly fail on Web Workers and in Node.
https://github.com/gpuweb/cts/blob/master/src/suites/cts/canvas/context_creation.spec.ts#L8

[dcerisano]

Thanks @kainino0x. Please let me know when I should pull.

Is there a way to turn on messages like:
t.skip('DOM is not available to create canvas element');

Really I would like to turn on all the messaging and logging there is - this is proving very challenging to debug in my head.

[kainino0x]

I don't think there is a way to do that right now in the cmdline runner, but there is a "debug" mode that is wired up elsewhere. I will wire it up into the cmdline runner.

[kainino0x]

Actually, the problem was I didn't make the cmdline runner understand the "skip" status (oops). Fixing that and wiring up the debug mode.

[kainino0x]

Ok, done and merged. Once you update, try running tools/run with --verbose --debug. (Though --debug will probably not be that helpful most of the time.) This should be much better for debugging.

I created a new cts-napi branch which currently is the same as master. I'm happy to take PRs in there anytime and I'll take care of any issues (e.g. in case there are any issues with running in browser) before merging that branch to master.

[dcerisano]

Just got back to this. Thanks for the last push.

I am getting some of the CTS tests to pass with the WebGPU-NAPI addon, but not happy with the current approach. The issue right now is that the WebGPU-NAPI addon is not fully compliant with the @webgpu spec.

Rather than bending CTS and @webgpu to fit the WebGPU-NAPI (current approach), I am thinking that making WebGPU-NAPI comply with @webgpu is a better approach. It is going to have to be done anyway as the official spec evolves.

This will mean PRs to @maierfelix's WebGPU-NAPI addon in addition to PR's here. I will ask @maierfelix to create a cts-napi branch in his repo.

This will pay off when I finally get around to the WebGPU-V8 extension. I want to be able to use the WebGPU-NAPI addon and CTS directly there with no modifications to the @webgpu spec.

If browser, NodeJS and V8 WebGPU implementations are all @webgpu compliant then the ultimate goal of portable WebGPU code between these three environments can be realized.

[kainino0x]

Sounds like a good plan! Thanks for the update!

[Kangz]

Closing this. The CTS runs well inside dawn_node.