Tracking issue for using WebGPU to render on the window of the desktop environment about deno

[crowlKats]

We will not add a dependency on winit or any other windowing system, the way we have it currently is the best compromise.

Regarding Canvas and related:
for OffscreenCanvas, I have a local branch that needs some further work. The idea is that UnsafeWindowSurface will extend OffscreenCavas.
We will not implement a webgl context or any webgl related APIs, however we likely will implemented 2dcontext via Vello.

[chirsz-ever]

For ImageBitmap :

• ImageBitmap is designed for immutable bitmap data stored in GPU (but not necessary, for example, Firefox do not store the data in GPU)¹², so it has no interface to get the underlying pixel data.
• The following functions use ImageBitmap:
  • createImageBitmap()
  • CanvasRenderingContext2D.drawImage()
  • CanvasRenderingContext2D.createPattern()
  • WebGLRenderingContext.tex(Sub)Image2D()
  • ImageBitmapRenderingContext.transferFromImageBitmap()
  • OffscreenCanvas.transferToImageBitmap()
  • GPUQueue.copyExternalImageToTexture()
• So we could make the following improvements:
  • Use pixel format other than RGBA, such as that in image::DynamicImage. This would save memory usage.
  • Use a GPUTexture as the underlying data resource when possible. This improves performance because you don't need to transfer data from and to the GPU.

Footnotes

1. https://stackoverflow.com/a/60033582 ↩

2. https://github.com/gpuweb/gpuweb/issues/1154 ↩

[crowlKats]

one you missed that uses ImageBitmap is https://gpuweb.github.io/gpuweb/#gpuimagecopyexternalimage

[chirsz-ever]

Why don't allow "none" colorSpaceConversion when createImageBitmap?

https://github.com/denoland/deno/blame/c5193556242117737fc3ba95ee3692b7831ca04a/ext/canvas/01_image.js#L307-L309

[chirsz-ever]

Is this a bug? I think the alpha value do not need to multiply 255. And why there need a is_not_premultiplied flag calculated from every pixels?

https://github.com/denoland/deno/blame/c5193556242117737fc3ba95ee3692b7831ca04a/ext/canvas/lib.rs#L86

[crowlKats]

• Use pixel format other than RGBA, such as that in image::DynamicImage. This would save memory usage.

sounds reasonable

• Use a GPUTexture as the underlying data resource when possible. This improves performance because you don't need to transfer data from and to the GPU.

I don't think this is viable, since to create a GPUTexture you need a GPUDevice, which is not feasible since a user could use a different GPUDevice, or same GPUDevice but with different features.

[chirsz-ever]

We could hold the GPUDevice object in a ImageBitmap object when the underlying resource is a GPUTexture. When we want to perform some operations, we first check whether the devices on both sides are the same, and then directly perform the operation, or copy the data to the CPU side and then transfer it to the target.

This optimization would not be implemented in the near future, so it is recorded here just for reference.