Minutes 2021 05 03

GPU Web 2021-05-03

Chair: Corentin / Jeff

Scribe: Corentin / Kai

Location: Google Meet

Tentative agenda

• Administrivia?
• Pluralize requestAdapters() and add GPUAdapter.isSoftware #1660
• copyExternalImageToTexture for HTMLVideoElement #1647
• GPUExternalTexture: API for video; WGSL type and functions #1666
• Add external_texture luminance functions #1681
• Behavior of importing canvases as GPUExternalTextures #1670
• What happens in configureSwapChain and on canvas resize? #1691
• (stretch) Add initial color spaces #1690
• Agenda for next meeting

Attendance

• Apple
  • Myles C. Maxfield
• Google
  • Austin Eng
  • Brandon Jones
  • Corentin Wallez
  • James Darpinian
  • Kai Ninomiya
  • Ken Russell
  • Shrek Shao
• Kings Distributed Systems
  • Hamada Gasmallah
• Microsoft
  • Rafael Cintron
• Mozilla
  • Dzmitry Malyshau
  • Jeff Gilbert
• Francois Daoust
• Francois Guthmann
• Michael Shannon
• Mehmet Oguz Derin
• Rick Battagline
• Timo de Kort

Administrivia

• FPWD and TAG review are go!

Pluralize requestAdapters() and add GPUAdapter.isSoftware #1660

• KN (via email): Get follow-up feedback from Myles, probably avoid extended discussion for today.
• MM: Discussed internally, view didn't change from last week. Opening paragraph talks about the editor's discussion saying the approach would be simpler. Care to elaborate?
• KN: It's difficult to express all the use cases of the API.
  • 1. would be valuable to have the application be able to request software adapter and fallback to other adapter. If you have a way to get the software adapter, then you can do it with multiple requestAdapter calls. We want to encourage people to use software for a better experience on the Web, so want to make it harder to not use software. Suggested an allowSoftware enum "never" "lastResort" "always" and more. Tried to find a way to express that with two boolean flags but couldn't figure it out. TL;DR it was complicated.
  • MM: Are the usecases listed in somewhere, would be nice to have them in the issue.
• MM: To explain why we don't want requestAdapters. One thing is that we don't want is websites getting a list and interrogating it. Another one is having requestAdapters makes it easier for browsers to do willful violation of the spec and return more then two adapters in a way that doesn't cause application breakage.
• CW: Can we have more details? For example node.js adding requestAdapterS is a willful violation but it doesn't matter.
  • MM: Node.js has a lot of non-standard APIs so that's not what the majority of the concern is.
  • BJ: There is value in guiding the ecosystem of node.js electron and Deno to have the right direction.
  • MM: Can have a non-normative note for these JS runtimes.
• CW: At this point I would like to go in writing it would be easier to ask for clarification.
• MM: Fine by us.
• KR: Worried about the wording of "willful violation" that put things in an antagonistic tone. We are trying to create an extensible API, if we design something that allows a plurality of use-cases in the future without changing the init code. Would be sad that we don't allow this in spec. Could be tested that we return only 2 in WPT, then the group can discuss in the future whether to extend and when. If we accept that init code is a hard problem and we need to design for extensibility, we'll be in a better place.
• KN: AI to create a new issue for the discussion and write down usecases

copyExternalImageToTexture for HTMLVideoElement #1647

• KN(email): I'm not sure this overload is useful (can use import instead). I'm proposing cutting it from the API for now.
• KN: Did copyEI2T only for canvas and ImageBitmap for now. Now going to video. Is it actually useful when we also have the import path? Two reasons
  • In general implementations will do a draw with a render pass and the application can do that with importTexture.
  • Video streams can resize and change over time, and the application would have to react to that, with the copy just being a 1-1 pixel mapping. ImportTexture makes it easier to not care about what happens in the video.
• JG: It should be easy to make a polyfill for copyEI2T for video, but would need to write it.
• MM: When using an external_texture you can sample with [0, 1] but also samplers with unnormalized coordinates.
  • KN: textureLoad is [0, N]
  • CW+JG: WebGPU doesn't have unnormalized coordinate textures.
• RC: Does it mean that we're going to have to specify importTexture completely for WebGPU v1?
  • KN: Yes, but you can polyfill in using a copy internally.
  • KN: Complicated to implement with all the craziness of videos.
• MM: So this PR says fat sampler, or the browser polyfilling fat sampler is the only way to get video in WebGPU.
• MM: Seems fine to not have it, plus can add it later.
• MM+KN: Fat sampler much more complicated than copyEI2T.
• MM+CW: So you can polyfill copyEI2T with the fat sampler but would need to add an additional submit.
• MM: In Metal performance shader it returns a command buffer that you can add to your queue submission.

GPUExternalTexture: API for video; WGSL type and functions #1666

• Info: Binding cost is 4 textures + 1 sampler + 1 uniform buffer. Read using textureLoad or textureSampleLevel (where level always = 0).
• KN(email): Does not package its own sampler, user provides their own. Means we cannot do post-conversion filtering if 0-copy. OK?
• No more Qs, feedback if anyone has it.
• KN: Adds the fat sampler just for video (canvas is separate). It adds the WGSL type and builtins. The external_texture is paired with a user-provided sampler with textureSampleLevel (with implicit level 0). This means that YUV planes samples will be done with the user-provided sampler.
  • KN: This prevents the browser from doing correct filtering for Y and UV because it doesn't know what the sampler does. If we wanted to do correct filtering, then we'd need to pass the sampler data in the GPUExternalTexture.
  • KN: That's the biggest open question. Do we want to change the design so that the texture and sampler are bundled together on GPUExternalTexture so the implementation can choose when it wants to implement the sampling.
  • MM: Can't implementation do a copy to implement the correct filtering and that's legal?
  • KN: Yes but that's an extra blit.
  • MM: Might be able to interrogate the sampler in the shader. Can't find it.
• JG: Would like to have that in writing with an example.
• KN: Will do.
• MM: So the question is would any implementation want to do a blit for correct filtering instead of a fat sampler that's slightly incorrect.
• JG: The fat sampler would naively linearly filter in luma/chroma space, not rgb. It’s closer than you would expect, but it’s still a different result.
• CW: Fat sampler contains a UBO including color conversion, transforms, etc. It could also include the sampling parameters (linear/nearest) so we could emulate the filtering inside the shader (load four values then lerp them).
• MM: Seems to need implementation experience.
• CW: In chromium today, resizing videos uses “incorrect” (pre-conversion) filtering and we have never gotten a bug report about this. So we don’t intend to “emulate” that post-conversion filtering.
• MM: So the cost of correctness would be performance.
• KN: A lot more shader code, ALUs, etc. If we wanted to add a "Fine" option we could add it later.
• MM: Could be a new WGSL type.
• RC: Could people use WebCodec to have the correct filtering?
• KN: We'd need to add something new for WebGPU for WebCodec anyway. If people want to do correct filtering then it's fine that they need to use WebCodec.
• RC: So colorspace conversion etc is in there.
• KN: Correct.
• JG: Filtering has a lot of precision issues, but if you have load, then it is 100% correct.
• CW: Good argument, the user can inline the filtering in their shader with textureLoad instead of the browser.
• JG: Compositor is incorrect anyway.
• RC: There's some correct filtering for YUV in D3D12. Users can also make things correct by making a copy with 1-1 ratio and then sampling.
• CW: Seems there's consensus.
  • DM:Not 100% equivalent with what the user can do. We can use instructions the user cannot, like gather.

Add external_texture luminance functions #1681

• KN(email): Complicated. Likely impossible to make portable+efficient. How important do people think this is? Can we postpone it?
• KN: This is actually complicated. Like what is the Y axis? If you need to convert between Y axises, then you need to load YUV go RGB and compute Y.
• KN: We could leave it undefined what kind of Y you get and you don't get consistent results across videos. So complicated and it's not clear how important it is.
• JG: Relatively strong user story: a lot of image processing stuff works purely in luminance space. What people want is an approximation of the luminance. For that user story, people what approximate luminance, inaccuracies are fine. It is like a proxy place for giving the Y plane.
• CW: So does this group want something like textureLoadLuminanceApproximation?
• JG: Don’t think the name needs to be too scary, result will be pretty close to what they expected.
• MM: So what if the application wants the real luminance? YUV -> RGB -> Y
• KN: Would do that, but the browser will have to do this if it wanted to give you the real luminance or not.
• KN: If all videos give you the same luminance then we could use that?
• JG+KN depends on the color space.
• MM: So for YUV video it gives you the Y channel, whatever it is, and for RGB it does some math.

Behavior of importing canvases as GPUExternalTextures #1670

• Q: Please review the investigation. Proposed to allow ONLY OffscreenCanvas (4+5), but would require everyone to implement it for 2d+webgl+webgl2 - OK? If not, discuss 4+2a vs 4+2b vs 4+3.

What happens in configureSwapChain and on canvas resize? #1691

• KN(email): No Q, just feedback if anyone has it.
• (Discussion about how WebGL works)

(stretch) Add initial color spaces #1690

• KN(email):
  • No Q, just feedback if anyone has it.
  • Info: Every time color values (canvas/video/image) are converted to raw values (GPUTexture), the raw values will be tagged with a color space, e.g. srgb, srgb-linear, display-p3, etc. List comes from CSS, full list TBD.
  • srgb is always the default (not sure yet if clamped-srgb or extended-srgb). Bytes written to texture are the same for unorm and unorm-srgb, but the latter allows you to sample linear values.
  • srgb-linear needed to decode images into linear space, e.g. for linear encoded float textures.

Agenda for next meeting

• CW: Carry over agenda, but other topics?
• Pluralize requestAdapters() and add GPUAdapter.isSoftware #1660
• Add external_texture luminance functions #1681
  • Possibly discuss precise version
• Behavior of importing canvases as GPUExternalTextures #1670
• What happens in configureSwapChain and on canvas resize? #1691
• (stretch) Add initial color spaces #1690