Minutes 2022 05 18

GPU Web 2022-05-18

Chair: Corentin

Scribe: Ken / Corentin / others

Location: Google Meet

Tentative agenda

• Administrivia
• CTS Update
• “Tacit Resolution” Queue
• Handling out-of-memory when resizing canvas backing store #2520
• WebGPURenderingContext should support preserveDrawingBuffer: true #2743
• Internationalization Metadata for Human-readable Messages #2780
• maxBuffersPlusVertexBuffersForVertexStage limit #2749
• Missing depth24unorm fomat? #2732
• Agenda for next meeting

Attendance

• Apple
  • Myles C. Maxfield
• Google
  • Austin Eng
  • Brandon Jones
  • Corentin Wallez
  • Kai Ninomiya
  • Ken Russell
  • Loko Kung
  • Stephen White
• Microsoft
  • Jesse Natalie
  • Rafael Cintron
• Mozilla
  • Jim Blandy
  • Kelsey Gilbert

Administrivia

• CW: W3C is interested in helping the WebGPU working group with developer support. What that means can take many shapes. After we resolve more of the big spec issues, or maybe in parallel/backburner - we can start talking about this. We have the WebGPU Github org, we have multiple domain names, etc. We’ll have to figure out what this support includes.

CTS Update

• KN: Personally been working on maintenance and linting. There have been ongoing contributions to add tests.
• CW: Please contribute and run tests in your browser.

“Tacit Resolution” Queue

• Make GPUShaderModuleCompilationHint.layout optional #2876
  • Raised by Dzmitry. On Naga/wgpu, would be interesting to have primitive type of pipelines in the hints as well.
  • This PR doesn't solve that - but does change the hints structure to make everything optional. In the future, can extend it with all the data we want. Developers can put everything they have in there. Solves the extensibility problem.
  • MM: ironically, this was the way it was originally. Then I received feedback on PR to make it non-optional.
  • CW: you were right :)
  • MM: reason - there's only one dictionary member, whole dictionary is optional, so no need to make the member optional. Agree, each field should be optional so they're independent.
  • CW: maybe next Monday you'll have been right all along :)
• Relax the null buffer check for zero-attribute vertex buffers in "valid to draw with GPURenderCommandsMixin" #2864
  • From Takahiro, while trying to implement something small.
  • In Dawn: render pipeline requires buffers with 0 attributes to still be set as data on render pass when doing draw call with render pipeline. Loading 0 data, seems silly.
  • Maybe spec doesn't need to validate the buffer's present if no vertex attributes used from that buffer.
  • People agreed to close the issue. If render pipeline says it needs a vertex buffer, even if 0 attributes, it could be for programmatic use cases.
  • Tacit resolution: keep spec as is.
  • KN: if we do find in the future this is annoying - we can change it. It'll be loosening some validation. But think this can always be worked around on app side.
• Should cullMode require "none" for points and lines? #2863
  • BJ: just a note in the spec. Cull mode doesn't do anything for these. Doesn't feel it's worth validating. None of the native APIs have any adverse effect if you set it to something else. Tacit resolution is to remove the note.
  • KG: sure.

Handling out-of-memory when resizing canvas backing store #2520

• KN: With #2821 the spec now explicitly mentions that out-of-memory errors can be generated in getCurrentTexture(). Is this good for V1? Or can we close it?
• KN: one situation where this might not work perfectly: if we put a limit on the megapixels of a canvas, we'd have to expose this as either validation or out-of-memory error. Similar to validation in createBuffer. No chance here to inject an extra error except OOM.
• Seems to me that errors from allocation should be the same from createTexure or getCurrentTexture, and that spec's fine here for V1.
• MM: so options are: if backing store too big but you still have memory, expose as OOM or new thing? Is that the question?
• KN: essentially - do we need to be able to expose something like that? Browsers being able to say something different than OOM? Allocation would be ~the same size as createTexture of the same size. Think rules shouldn't be different between textures and canvas textures, so don't need something new here.
• GT: today in WebGL you can find out what the limit is. Here you need to binary search for it.
• KN: I think if we do this, browsers will have to not add extra limits. Should only raise OOM when we actually ran out of memory. No additional constraints here. Today, limit is texture size on the device.
• KG: if no theories about why we might need more restrictions than that, go with this.
• Discussion about WebGL, and Chrome's implementation.
• BJ: silently scaling things down if you allocate a WebGL back buffer above a certain size. Just because of driver issues?
• KR: I tried removing the size limit in Chrome’s implementation. Some WebGL conformance tests make large canvases. Removing the constraint, we saw crashes on every platform when the actual OpenGL driver ran out of memory. We’ve started moving implementations off OpenGL onto ANGLE since I last tested this. I think this is a driver robustness issue, and we should proceed assuming we can use the max texture size limit in WebGPU.
• KG: for what it's worth Gecko has not had the same trouble. But also, why WebGL is different [from webgpu] - what happens when you ask for a too-large buffer. Impl-defined there. We give you a buffer back, and if you blindly render into it - will give you a smaller view of things. Ask for 10k x 10k and max texture size is 8k or less - scaled down, but will still succeed allocation.
• BJ: we're talking about "framebuffer" here - that's the reason for the conflation between "buffer" and "texture" here. Don't have the auto-scale-down option in WebGPU because you have to manually allocate e.g. depth/stencil buffers at the same size.
• CW: WebGL's backbuffer as one big package is one big difference. Also in WebGL / OpenGL, OOM is not really recoverable, but in WebGPU and underlying APIs it is. OK for things to be different in this case.
• MM: Ken do you remember what the limit is?
• KR: it was designed for the 5K iMacs, so around 8 megapixels.
• MM: no indication this kind of limit will be necessary, so this seems reasonable. If we end up needing it later we can come back to this.
• KG: generally agree. In WebGL I wouldn't say OOMs are unrecoverable. Chrome's had more issues than Firefox has. But Chrome's getting better in this area. Tricky but doable.
• CW: hearing no objections to going with that. Resolved as - keep it like that. If new concern comes up we can address it.

WebGPURenderingContext should support preserveDrawingBuffer: true #2743

• KN: made a proposal, came out pretty large. Summary:
• KN: as discussed last week - allows reading back from the canvas after present. Rather than clearing the JS-observable contents of the canvas at the frame boundary - we preserve it at frame boundary, and not cleared until getCurrentTexture.
• KN: It's not preserveDrawingBuffer, which would always preserve the contents. It doesn't add preserveDrawingBuffer. It holds onto the canvas contents. We terminate write access - destroy the GpuTexture, image becomes read-only. Then has shared read-only access between compositor and the page. Will read back the same image being concurrently used by the compositor.
• KN: spec doesn't describe it this way - describes these as copies, and snapshots.
• KN: but it's designed so that copies can be elided as moves. Alpha clear which happens on opaque canvases can be done in-place after presentation, because at that point it's no longer observable whether alpha channel's cleared.
• KN: e.g. on Mac: until frame boundary, operating on texture. When presented: clear alpha channel of the same buffer / working memory as before. Terminate write access. Take the cleared image, that's what's used for compositing and readbacks after presentation.
• KN: readbacks don't need anything done eagerly. Can do clear upon readback, but no extra work to enable readbacks. Can enable them all the time.
• KN: main concern: can you actually implement shared read-only access between page and compositor? Started discussions internally for Chrome. D3D might make this slightly complicated - requires using fences added in later Windows 10 versions. (I don't fully understand that, but hopefully folks on this call can understand their own implementation constraints.)
• CW: on D3D12 side - 2 means of synchronizing access to OS textures. Keyed mutexes, and D3D12 fences. Fences allow concurrent read access. Keyed mutexes don't. Fences were introduced soon after Windows 10. If Chrom required D3D12 fences, we'd lose ~2% of Win10 users. Acceptable to us. If WebGPU compat, they can always get that.
• BJ: with process just described - doing alpha clear on frame boundary - readbacks will have alpha channel cleared. Does that mean we get different readback results depending on before/after frame boundary?
• KN: I changed compositingAlphaMode to alphaMode. Always affects readback before/after presentation. Might have to do a clear if mid-frame. Avoid using opaque if you want to avoid it.
• BJ: if I do a readback mid-frame does that affect the canvas?
• KN: no, it makes a copy.
• BJ: won't incur the copy if alphaMode=transparent?
• KN: correct.
• BJ: OK.
• JN: clarifying D3D12 fences. Have been there since day 1. Later Windows addition was equivalent fences for D3D11. No OS version differences here.
• RC: think the reason Chromium team is mentioning this is because of use of D3D11. Not sure why the complexity applies, but I'm probably missing context in the conversation. Discuss more offline?
• JN: sounds good.
• KG: this proposal's interesting - I made something like this for WebGL at some point. How I implemented toDataUrl in Gecko at one point. If you hadn't drawn to new buffer yet, it gave you the old buffer. But if you had drawn to it, gave you the new buffer's contents. On at least some platforms - and this is before what we did on Android with SurfaceTextures, where it's hard to get the buffer back from the compositor -
• CW: can't you use AHardwareBuffer?
• KG: not to my understanding.
• KN: on WebGL not sure what implications are for preserveDrawingBuffer, but similar techniques could be used for readbacks there. Make the copy later when there's a draw call. If whole thing was cleared, don't need to make the copy. Don't think WebGL implements this optimization but it could.
• MM: chatting with one of GPU engineers. He's saying - if you have a MtlDrawable that gives you MtlTexture, you do the rendering, and you present - it's legal to keep a handle to that texture and read back from it only if framebufferOnly is set to false. If set to true, you can't do that.
• CW: if framebufferOnly=true, can you never read back from the texture?
• MM: yes.
• CW: could use pixel local storage maybe? But that's a hack, not universally available.
• MM: I don't want to set a flag at the beginning of time and make everything slower. If modifying fragment shaders to have side channel, won't work.
• KG: directions I like: 1) Kai's structure, 2) preserveDrawingBuffer togglable live - would handle many use cases. 3) async ImageData snapshot on canvas element.
• KR: what about ImageBitmap instead? Could be kept GPU-side.
• MM: "give me a snapshot" sounds reasonable and totally implementable on the Metal side. More intent's captured in that - supposed to capture a screenshot.
• KN: couple comments. 1) we could add a new API if we really think it's necessary to diverge from existing stuff. But those APIs already exist. async's not necessarily the right solution. Lots of thoughts about async creation of ImageBitmaps. If GPU-side, doesn't need to be async. Can go from WebGPU canvas to WebGL canvas. Just enqueues a GPU operation.
• KN: don't think we'd want to force a JS async in here.
• KN: 2) doesn't seem necessary to diverge from existing API. Have a library that wants to take screenshot of page. Library could be modified to call this thing and async readback the data. But doesn't seem necessary to me.
• GT: There's already toBlob which is already async. Don't see why a new API helps - you have no control over the thing doing the rendering. If I did have the control I wouldn't need this new thing. Can't force the other thing you're trying to snapshot, to update.
• KG: intent here - you ask for a snapshot. Not because waiting for next draw - but make some IPC calls to ask the compositor to screenshot this. Do some heavy lifting. Reason that occurred to me - ImageBitmaps could make it non-async, but not even on Firefox's roadmap to make ImageBitmap non-CPU-side. Becomes a bigger request.
• KN: it's true even with toDataUrl. Assume we want toDataUrl to work within a frame. Discussion's about next frame, after it's presented. Still need CPU readback. Ideally don't incur CPU readbacks. Want the API surface, have user express that they want that.
• KG: problem with making it optimizable - will work well if optimized, but if not, will work extremely poorly.
• KN: not really adding more. Just a new canvas context type. Still the same as the other canvases.
• KG: if goal's to be able to snapshot a canvas - maybe just add the concept of snapshotting a canvas rather than modifying other things' behavior.
• CW: what's the set of things that would interact with this? toDataUrl, createImageBitmap.
• KN: texImage2D
• KG: drawimage
• CW: only programmatic one - toDataUrl.
• KR: non-programmatic: printing, "Save Image" context menu for canvases.
• CW: sounds like toDataUrl is the only one.
• KG: in this case - I think toDataUrl's a disjoint use case. "Take a screenshot of this" is different from "I want my copies to be efficient and fast". Draw into a side buffer, or you rendered it a while ago (Google Maps) and want to print it. Be careful about trying to be too perfect here.
• CW: KG you are concerned about the semantic of these 5-6 operations that can read the WebGPU back buffer? Others are things that with Kai's proposal work well. Only one's toDataUrl. No concern about the others?
• KG: don't think so.
• CW: OK.
• KN: you said before you still need to read back to go WebGPU to WebGL.
• KG: right now canvas-to-canvas is CPU bound. Have a todo about this, but not high priority.
• CW: does it round-trip to content process?
• KG: yes. Turns out it's easy to think about use cases for these, but don't happen that often. Even though Chrome optimizes these, and you'd think people would develop/test on Chrome and they'd go slow on Firefox - not common use case. I'm concerned with satisfying ability to do a thing. Get data back after we've drawn it, in a non-cooperative way.
• SW: text and texture atlases.
• KG: text's not GPU-able - all rendered on the CPU.
• SW: point - people draw glyphs into 2D canvas, then upload to WebGL texture.
• KG: haven't seen that as a perf issue. Usually do text run caching.
• MM: impl I've seen, creates atlas in 3D engine, not 2D engine. tiny canvas for single glyph, then copy into 3D atlas one-by-one.
• KG: it's what we do in WebRender.
• CW: so - Firefox works? Doesn't have GPU-based canvas-to-canvas copies. Doing things synchronously - unpalatable, would add more sync IPCs.
• CW: for snapshotting - can be toDataUrlAsync, also createImageBitmap (which is async). How would people take WebGPU data into WebGL/2D canvas?
• KR: texImage2D, drawImage.
• KN: seems readback-after-presentation doesn't affect this. Things will work similarly to other canvas types. Are we wanting to revisit that previous decision?
• KG: I'd like our solution here to be simpler. Simplest thing in this case - a toggle-able preserveDrawingBuffer. I have to think about this. Want to say - not all of these paths have to be fast.
• CW: since we wanted to support these use cases, and since proposal doesn't add specific overhead on CPU readbacks compared to any other proposal - seems fine. maybe we say it's fine, can't use toDataUrl. Can copy to 2D canvas.
• KG: not sure where toDataUrl problem came from. If it came from something I said,
• CW: Sorry, misunderstood.
• KR: Would like to advocate for keeping things simple. In WebGL, preserveDrawingBuffer came from OpenGL-flavored window system bindings. Hoping for Metal framebufferOnly we can find a solution in the future. But want to keep things simple now.
• MM: previous meeting - agreed that framebufferOnly flag will be unconditionally set to false, so now we can take snapshots after presentation. Later if we add some extension to allow framebufferOnly=true, that'd intentionally break APIs and be opt-in.
• KG: concerned that's like preserveDrawingBuffer:true in practice, but need to thin about it more.
• CW: Kai's proposal similar to preserveDrawingBuffer:true, but no cost to app that's not doing readbacks. Best of both worlds.
• KN: difference with preserveDrawingBuffer is that it doesn't preserve it until the next frame. Gets you everything else preserveDrawingBuffer did.
• MM: if someone wants that - think we should resolve they do it themselves. We should intentionally not support that use case.
• KN / CW: agree.
• KG: if on Metal you wanted framebufferOnly=true - think on Firefox we really do give up access to that buffer (on Android). Think we must be able to take snapshot of it somehow, but involved IPC process.
• CW: let's keep it at that. Work it out offline. If no concerns - if implementable without crazy hoops - would like to resolve this offline. Have talked about this 3 meetings.
• KN: these were the last canvas issues for V1! That should be it.
• KG: I'd be fine with this being post-V1, given how much we've not supported this in WebGL. In practice, think "acceptable loss".
• KN: WebGL's had preserveDrawingBuffer forever.
• CW: let's discuss offline.

Internationalization Metadata for Human-readable Messages #2780

maxBuffersPlusVertexBuffersForVertexStage limit #2749

Missing depth24unorm fomat? #2732

Agenda for next meeting

• CW, pre-meeting: Apple implementation issues next now that canvas is good.
• CW: What about trying to make offline progress on smaller issues we didn't cover today?
  • MM: think we should set the agenda based on whether offline resolution actually happens.
  • CW: carrying forward items here, and skip them next time if already resolved?
  • MM: would like to know which issues you're describing (written down) when you mention WebKit implementation issues.
  • CW: can send them to you via email now.
• CW: please add more items here.