Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add config to enable/disable mipmaps for raster layers. #1853

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

kircher1
Copy link
Contributor

Launch Checklist

This adds a config option to disable or enable mipmaps on raster layers. The default is mipmaps enabled.

For clients that don't gain much from having mipmaps generated (eg clients that don't allow tilting the view), then disabling mipmaps will get rid of any potential overhead from the mip level generation that GL is performing on every raster texture. And there are some subtle visual aspects where not using mip filtered textures could be desirable.

Note, another option for this sort of change is to add a style property to the raster data source which controls if mipmaps are enabled or not, rather than having a global config.

  • Confirm your changes do not include backports from Mapbox projects (unless with compliant license) - if you are not sure about this, please ask!
  • Briefly describe the changes in this PR.
  • Link to related issues.
  • Write tests for all new functionality.
  • Document any changes to public APIs.
  • Manually test the debug page.
  • Add an entry to CHANGELOG.md under the ## main section.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 18, 2022

Bundle size report:

Size Change: +131 B
Total Size Before: 206 kB
Total Size After: 206 kB

Output file Before After Change
maplibre-gl.js 197 kB 197 kB +131 B
maplibre-gl.css 9.09 kB 9.09 kB 0 B
ℹ️ View Details
Source file Before After Change
src/index.ts 883 B 916 B +33 B
src/source/raster_tile_source.ts 912 B 944 B +32 B
src/data/bucket/pattern_attributes.ts 111 B 140 B +29 B
src/render/texture.ts 679 B 705 B +26 B
src/util/config.ts 83 B 104 B +21 B
node_modules/murmurhash-js/murmurhash2_gc.js 244 B 265 B +21 B
node_modules/earcut/src/earcut.js 2.7 kB 2.72 kB +20 B
src/util/classify_rings.ts 246 B 258 B +12 B
node_modules/gl-matrix/esm/vec2.js 280 B 291 B +11 B
src/style/properties.ts 1.9 kB 1.91 kB +8 B
src/util/global_worker_pool.ts 319 B 327 B +8 B
src/render/draw_background.ts 572 B 580 B +8 B
src/data/bucket/heatmap_bucket.ts 78 B 85 B +7 B
src/util/image.ts 716 B 723 B +7 B
src/render/draw_hillshade.ts 1.15 kB 1.16 kB +7 B
src/render/draw_heatmap.ts 1.04 kB 1.04 kB +7 B
src/style-spec/expression/types.ts 512 B 517 B +5 B
src/render/uniform_binding.ts 643 B 648 B +5 B
src/data/pos_attributes.ts 80 B 85 B +5 B
src/ui/hash.ts 935 B 940 B +5 B
src/style-spec/expression/types/collator.ts 204 B 208 B +4 B
src/data/bucket/circle_bucket.ts 968 B 972 B +4 B
src/style/query_utils.ts 483 B 487 B +4 B
src/render/program/debug_program.ts 192 B 196 B +4 B
src/render/program/symbol_program.ts 1.29 kB 1.3 kB +4 B
src/util/webp_supported.ts 372 B 375 B +3 B
src/data/bucket/circle_attributes.ts 92 B 95 B +3 B
node_modules/gl-matrix/esm/vec4.js 393 B 396 B +3 B
src/util/color_ramp.ts 433 B 436 B +3 B
src/data/bucket/pattern_bucket_features.ts 324 B 327 B +3 B
src/data/raster_bounds_attributes.ts 93 B 96 B +3 B
src/util/web_worker.ts 39 B 42 B +3 B
src/shaders/fill.vertex.glsl.g.ts 171 B 174 B +3 B
src/shaders/hillshade.vertex.glsl.g.ts 136 B 139 B +3 B
src/render/program/background_program.ts 473 B 476 B +3 B
src/gl/index_buffer.ts 348 B 351 B +3 B
src/util/script_detection.ts 1.65 kB 1.65 kB +2 B
src/data/segment.ts 449 B 451 B +2 B
node_modules/gl-matrix/esm/common.js 180 B 182 B +2 B
src/shaders/circle.vertex.glsl.g.ts 557 B 559 B +2 B
src/shaders/fill_outline_pattern.fragment.glsl.g.ts 418 B 420 B +2 B
src/shaders/fill_pattern.vertex.glsl.g.ts 387 B 389 B +2 B
src/shaders/symbol_icon.vertex.glsl.g.ts 947 B 949 B +2 B
src/shaders/terrain_coords.fragment.glsl.g.ts 168 B 170 B +2 B
src/render/program/circle_program.ts 454 B 456 B +2 B
src/ui/handler/keyboard.ts 569 B 571 B +2 B
src/ui/control/geolocate_control.ts 2.24 kB 2.24 kB +2 B
src/util/ajax.ts 2.39 kB 2.39 kB +1 B
src/style-spec/expression/scope.ts 194 B 195 B +1 B
src/style-spec/util/padding.ts 257 B 258 B +1 B
src/style-spec/expression/definitions/assertion.ts 624 B 625 B +1 B
src/style-spec/expression/definitions/case.ts 459 B 460 B +1 B
src/style-spec/expression/definitions/number_format.ts 529 B 530 B +1 B
src/style-spec/expression/index.ts 1.66 kB 1.66 kB +1 B
src/style-spec/feature_filter/index.ts 889 B 890 B +1 B
src/style-spec/validate/validate_boolean.ts 122 B 123 B +1 B
src/symbol/symbol_layout.ts 3.81 kB 3.81 kB +1 B
src/style/load_glyph_range.ts 222 B 223 B +1 B
src/style/light.ts 559 B 560 B +1 B
src/source/vector_tile_source.ts 1.1 kB 1.1 kB +1 B
src/util/offscreen_canvas_supported.ts 154 B 155 B +1 B
src/source/geojson_source.ts 1.32 kB 1.32 kB +1 B
src/source/canvas_source.ts 1.02 kB 1.02 kB +1 B
src/source/source.ts 351 B 352 B +1 B
src/source/tile_cache.ts 555 B 556 B +1 B
src/util/worker_pool.ts 419 B 420 B +1 B
src/symbol/grid_index.ts 1.35 kB 1.35 kB +1 B
src/shaders/_prelude.vertex.glsl.g.ts 956 B 957 B +1 B
src/shaders/background.fragment.glsl.g.ts 137 B 138 B +1 B
src/shaders/heatmap.fragment.glsl.g.ts 266 B 267 B +1 B
src/shaders/heatmap_texture.vertex.glsl.g.ts 143 B 144 B +1 B
src/shaders/collision_box.vertex.glsl.g.ts 315 B 316 B +1 B
src/shaders/collision_circle.fragment.glsl.g.ts 258 B 259 B +1 B
src/shaders/debug.fragment.glsl.g.ts 147 B 148 B +1 B
src/shaders/fill_extrusion.vertex.glsl.g.ts 622 B 623 B +1 B
src/shaders/fill_extrusion_pattern.fragment.glsl.g.ts 415 B 416 B +1 B
src/shaders/hillshade_prepare.fragment.glsl.g.ts 500 B 501 B +1 B
src/shaders/line_gradient.fragment.glsl.g.ts 350 B 351 B +1 B
src/shaders/line_pattern.vertex.glsl.g.ts 787 B 788 B +1 B
src/shaders/line_sdf.vertex.glsl.g.ts 808 B 809 B +1 B
src/render/program/pattern.ts 615 B 616 B +1 B
src/render/program/heatmap_program.ts 558 B 559 B +1 B
src/gl/vertex_buffer.ts 608 B 609 B +1 B
src/gl/depth_mode.ts 135 B 136 B +1 B
src/render/draw_symbol.ts 2.61 kB 2.61 kB +1 B
src/render/draw_line.ts 1.05 kB 1.05 kB +1 B
src/render/draw_custom.ts 337 B 338 B +1 B
src/util/primitives.ts 728 B 729 B +1 B
src/ui/handler/touch_zoom_rotate.ts 1.03 kB 1.03 kB +1 B
src/ui/handler/tap_drag_zoom.ts 482 B 483 B +1 B
src/ui/camera.ts 3.41 kB 3.41 kB +1 B
src/gl/render_pool.ts 583 B 584 B +1 B
src/ui/marker.ts 2.87 kB 2.87 kB +1 B
src/ui/control/terrain_control.ts 418 B 419 B +1 B
src/style-spec/util/color.ts 317 B 316 B -1 B
src/style-spec/util/interpolate.ts 232 B 231 B -1 B
src/style-spec/expression/definitions/format.ts 650 B 649 B -1 B
src/style-spec/validate/validate_image.ts 62 B 61 B -1 B
src/data/array_types.g.ts 2.82 kB 2.82 kB -1 B
src/data/program_configuration.ts 2.62 kB 2.61 kB -1 B
src/data/extent.ts 32 B 31 B -1 B
src/render/glyph_manager.ts 947 B 946 B -1 B
src/render/line_atlas.ts 984 B 983 B -1 B
src/source/raster_dem_tile_source.ts 971 B 970 B -1 B
src/source/video_source.ts 876 B 875 B -1 B
src/source/source_cache.ts 3.99 kB 3.99 kB -1 B
src/style-spec/diff.ts 1.54 kB 1.54 kB -1 B
src/symbol/path_interpolator.ts 311 B 310 B -1 B
src/style/pauseable_placement.ts 598 B 597 B -1 B
src/symbol/cross_tile_symbol_index.ts 1.37 kB 1.37 kB -1 B
src/shaders/line_sdf.fragment.glsl.g.ts 460 B 459 B -1 B
src/shaders/symbol_sdf.fragment.glsl.g.ts 554 B 553 B -1 B
src/render/program/terrain_program.ts 695 B 694 B -1 B
src/render/program/fill_program.ts 570 B 569 B -1 B
src/render/program/collision_program.ts 724 B 723 B -1 B
src/gl/value.ts 1.09 kB 1.09 kB -1 B
src/gl/cull_face_mode.ts 154 B 153 B -1 B
src/render/painter.ts 3.76 kB 3.76 kB -1 B
src/geo/edge_insets.ts 431 B 430 B -1 B
src/geo/transform.ts 4.54 kB 4.54 kB -1 B
src/ui/handler/touch_pan.ts 541 B 540 B -1 B
src/ui/handler/shim/drag_rotate.ts 179 B 178 B -1 B
src/render/terrain.ts 2.08 kB 2.08 kB -1 B
src/ui/control/scale_control.ts 733 B 732 B -1 B
src/ui/control/fullscreen_control.ts 783 B 782 B -1 B
src/util/tile_request_cache.ts 934 B 932 B -2 B
src/data/feature_position_map.ts 614 B 612 B -2 B
node_modules/gl-matrix/esm/mat4.js 2.69 kB 2.69 kB -2 B
node_modules/@mapbox/tiny-sdf/index.js 1.1 kB 1.1 kB -2 B
src/source/image_source.ts 1.11 kB 1.11 kB -2 B
src/source/query_features.ts 1.22 kB 1.21 kB -2 B
src/shaders/_prelude.fragment.glsl.g.ts 121 B 119 B -2 B
src/shaders/circle.fragment.glsl.g.ts 410 B 408 B -2 B
src/shaders/collision_box.fragment.glsl.g.ts 148 B 146 B -2 B
src/shaders/collision_circle.vertex.glsl.g.ts 547 B 545 B -2 B
src/shaders/fill.fragment.glsl.g.ts 178 B 176 B -2 B
src/shaders/fill_outline_pattern.vertex.glsl.g.ts 415 B 413 B -2 B
src/shaders/fill_extrusion.fragment.glsl.g.ts 119 B 117 B -2 B
src/shaders/hillshade.fragment.glsl.g.ts 555 B 553 B -2 B
src/shaders/terrain.fragment.glsl.g.ts 112 B 110 B -2 B
src/render/program.ts 1.15 kB 1.15 kB -2 B
src/gl/stencil_mode.ts 150 B 148 B -2 B
src/ui/handler_manager.ts 2.46 kB 2.46 kB -2 B
src/ui/map.ts 7.16 kB 7.16 kB -2 B
src/style-spec/expression/types/formatted.ts 265 B 262 B -3 B
src/style/style_layer/circle_style_layer_properties.g.ts 230 B 227 B -3 B
node_modules/gl-matrix/esm/vec3.js 850 B 847 B -3 B
src/shaders/background.vertex.glsl.g.ts 106 B 103 B -3 B
src/shaders/heatmap.vertex.glsl.g.ts 369 B 366 B -3 B
src/shaders/fill_extrusion_pattern.vertex.glsl.g.ts 829 B 826 B -3 B
src/shaders/line_gradient.vertex.glsl.g.ts 739 B 736 B -3 B
src/render/program/clipping_mask_program.ts 106 B 103 B -3 B
src/render/draw_fill.ts 974 B 971 B -3 B
src/util/throttle.ts 145 B 142 B -3 B
src/util/is_char_in_unicode_block.ts 880 B 876 B -4 B
src/style/evaluation_parameters.ts 391 B 387 B -4 B
src/shaders/terrain.vertex.glsl.g.ts 222 B 218 B -4 B
src/util/util.ts 1.97 kB 1.96 kB -5 B
src/render/draw_circle.ts 620 B 615 B -5 B
node_modules/csscolorparser/csscolorparser.js 2.06 kB 2.05 kB -6 B
src/render/draw_fill_extrusion.ts 795 B 789 B -6 B
src/style/style_layer/hillshade_style_layer_properties.g.ts 164 B 157 B -7 B
src/render/draw_debug.ts 1.37 kB 1.37 kB -7 B
node_modules/gl-matrix/esm/quat.js 154 B 146 B -8 B
src/render/draw_raster.ts 1.05 kB 1.04 kB -8 B
src/style/style_layer/heatmap_style_layer_properties.g.ts 141 B 131 B -10 B
src/data/bucket/fill_attributes.ts 112 B 98 B -14 B
node_modules/quickselect/quickselect.js 385 B 366 B -19 B
src/render/program/program_uniforms.ts 960 B 940 B -20 B
node_modules/murmurhash-js/murmurhash3_gc.js 383 B 345 B -38 B

@HarelM
Copy link
Member

HarelM commented Nov 28, 2022

If this is something that should be done similarly in native and should be respected there as well I would consider using a style spec property for raster layer, I think, I'm not sure...
I would prefer if someone else would look at this as I'm not very familiar with the mipmap concept...

@kircher1
Copy link
Contributor Author

kircher1 commented Dec 2, 2022

@HarelM, is there someone we can pull into the request that can comment on the rendering aspect of this change?

I think using the style (and having a corresponding implementation in native) is a more involved approach but certainly doable. For that, I would suggest adding a boolean style property to the raster source that configures whether it generates mip maps or not. (With that approach, the internal changes to the Texture class would still look the same.)

@HarelM
Copy link
Member

HarelM commented Dec 2, 2022

@JannikGM are you familiar with the mipmap here? Can you take a look at this PR by any chance?

@JannikGM
Copy link
Contributor

JannikGM commented Dec 5, 2022

I'm not sure if this PR is beneficial; I don't follow the argument in the PR description.

For clients that don't gain much from having mipmaps generated (eg clients that don't allow tilting the view)..

This is not what mipmapping is for.

You must be thinking of anistropic filtering (which is related to mipmapping, but not mipmapping).

I'll mention the basics here:

  • mipmapping solves aliasing issues (zoom). imagine you have a 128x128 pixel image, and want to display it at 32x32 pixels: You'll have to pick every fifth pixel along each edge, but then you are discarding 4/5 of the image. As pixels you sample are further apart, the pixels you choose have a severe impact on image quality - that's a problem. You are also using more memory bandwidth as you have to load a 100x100 image, even if you only need 20x20 pixels from it - that's a problem.
    Mipmapping solves these problems, by taking the original 128x128 image and generating a 64x64 image (averaging 2x2 pixel blocks into a new pixel), then taking that 64x64 image and generating a 32x32 image; this repeats until you have a single 1x1 image. There is a slight memory overhead associated with storing these texture variations, but it's only taking 33% more memory than the original image would have taken (typically negible).
    At rendering the mipmap image which most closely resembles the display size is used. This means this is especially useful when looking at images top-down (such as raster maps).
    So, mipmapping actually has a performance benefit due to better bandwidth use (smaller image to load from RAM while rendering) + there's a visual quality improvement as you avoid aliasing issues.
    Mipmapping might not be wanted for things such as lookup tables where you want to look up exact pixels. It might also not be wanted for specific art-styles (such as retro/pixel styles).

    Raster tiles themselves might even be considered a form of mipmapping; so in maplibre, mipmapping might not be as important (as you only really need 2-3 mipmap levels for most tiles).

  • anistropic filtering solves issues where there is variation of zoom level in areas of an image (this happens at tilted angles, when some parts of the image are further away [smaller] than others).
    Imagine you have an 128x128 image that is displayed with 32 pixels near the top edge, but 128 pixels near bottom edge; the side edges are still about 64 pixels high; with mipmapping you'd have to find a compromise which image you'd pick (128x128? 64x64? 32x32?). Anistropic filtering sort-of solves, this by not only turning the 128x128 image into 64x64, 32x32, .. but also generating 128x64, 128x32, 128x16, 128x1; 64x64; 64x32...; 64x1; ... and 64x128, 32x128, ... this means you might need a lot more memory (300% of additional memory compared to the original image size).
    Anistropic filtering can also be more expensive during rendering and/or might take a lot of additional memory - it depends on wether the images are stored in memory in addition to traditional mipmaps or if the GPU dynamically generates these samples on the fly during rendering (typically done, so you can also handle diagonal distortion + control quality by setting the number of samples).

There are more details for each of these techniques and variations in how they can be used. I'd also suggest to look at the images on Wikipedia, as they nicely show these techniques (except that there's no example for zooming on a mipmap and their mipmap example is pretty much a case for anistropic filtering):

To me, it looks like mipmapping in maplibre also always enables anistropic filtering (if supported).
This, indeed, only makes sense for tilted views.

..then disabling mipmaps will get rid of any potential overhead from the mip level generation that GL is performing on every raster texture.

There still shouldn't be any noticable overhead with most GL implementations.

Mipmap generation is started using gl.generateMipmap(gl.TEXTURE_2D); which isn't handled by the JS world (which is generally slow).
Instead gl.generateMipmap typically takes minimal CPU effort because often only the (very fast) memory allocator in the GPU driver allocates space for the mipmaps, and then a couple of draw-calls are added to the GPU command queue.
The GPU, asynchronously to the CPU, will then render the mipmap images using linear filtering, often using a hardware blitter specifically for purposes like this.

From the application perspective, this should be free, only delaying the first frame by a fraction of a millisecond.
Because the GPU runs asynchronously (for the most part) and because frames are typically on a fixed framerate, it doesn't matter much if you are a fraction of a millisecond sooner or later... unless you were already close to being able to maintain the stable framerate.

The only cases where I'd imagine mipmap generation to be problematic are:

  • anistropic filtering of huge raster tiles (which should not exist?).
  • software rendering, where the mipmaps are generated on the CPU.

So I'd guess that this saves a neglible amount of memory, saves a tiny bit of setup time for each raster tile, but then adds additional cost during rendering while also degrading visual quality.

What can be done probably, is to limit mipmap levels to the required ranges (as most tiles are only displayed somewhere between 50% and 200% of their original size; I think?). I did not check if this is already being done.

Did you notice any real world performance issues with mipmap generations @kircher1 ? What hardware/software were you running on?


Edit

And there are some subtle visual aspects where not using mip filtered textures could be desirable.

Yes, something like a retro/pixelated look. But maplibre layers typically have edge smoothing and even for a pixel style you'd want edge smoothing in your raster tile to get a more consistent look across different devices.

@kircher1
Copy link
Contributor Author

kircher1 commented Dec 5, 2022

Re tilting: If the use case of MapLibre is a top-down map that doesn't allow tilt, the effect of mipmap is subtle on raster tiles. Like you say, the raster layer acts like a mipmap itself. A tilted view is the classic use case for mipmaps because texture aliasing is so apparent in oblique views.

e.g., wiki entry you linked to on mipmaps:
image

Aniso helps too but mipmaps are primarily the thing that is mitigating the aliasing.

Mipmaps and aniso are certainly the right tools to prevent texture aliasing, but if you use MapLibre in a way that doesn't allow tilting the view, then texture aliasing isn't a concern.

Re perf benefits: The memory overhead for the mip level limits to 1/3 of the original raw texture size. So, for a 256x256 RGBA tile (256 * 256 * 4 bytes) = 250KB, the additional memory overhead is ~83KB; Load a thousand raster tile textures, that's ~80MB of memory. Probably not going to be the tipping point of an OOM, but also not small.

The computation of the mipmap also has overhead, and until recent versions of Chromium this was a CPU side cost, which ate into javascript main thread time and could lead to jankiness: https://chromium-review.googlesource.com/c/angle/angle/+/3406643

Newer versions of Chromium defer the work to the GPU, where it can be done more efficiently. I don't know how other browsers handle it, but in the worst case, 1) you have a map that doesn't benefit from mipmaps, and 2) your user has no GPU acceleration. They pay the cost of generating mips on their CPU, wasting cycles that aren't needed.

@kircher1
Copy link
Contributor Author

kircher1 commented Dec 5, 2022

Also, there's a visual aspect to this, where having the mips enabled all the time can affect topdown views subtly, making them more blurry in certain zoom levels. I think this is because the GPU starts to sample the second miplevel instead of sticking to the highest miplevel. That is undesired in a page we have. What would be better there, would be a regular linear filter, which is slightly sharper (i.e. we don't need a mipmap) #1828 (reply in thread)

@JannikGM
Copy link
Contributor

JannikGM commented Dec 6, 2022

#1828 (reply in thread)

I wish this was part of the PR description. It provides some good context.

Aniso helps too but mipmaps are primarily the thing that is mitigating the aliasing.

I agree, but mipmapping at oblique angles, while avoiding the aliasing, usually also just leads to a blurry mess (without anistropic filtering). Head-on, mipmaps should always improve visual quality (if configured correctly).

Load a thousand raster tile textures, that's ~80MB of memory. Probably not going to be the tipping point of an OOM, but also not small.

Fair enough.

The computation of the mipmap also has overhead, and until recent versions of Chromium this was a CPU side cost[...]

Yikes! Being from the embedded world, I wasn't aware WebGL implementations were still this poor in this regard.
This is a convincing argument.

Like you say, the raster layer acts like a mipmap itself.

Let me preface this by saying that I have little experience with raster-tiles in maplibre.

But I should add that this also very much depends on the display resolution and underscaling / overscaling of the tiles.

Obviously a zoomed out map on a 4k display shouldn't be using a 0/0/0 256x256 raster tile.
Mipmapping is also quite useless then.
I think this is a common case, so I'm starting to lean toward disabling mipmapping.
(I'm not sure if you can force mapbox to use a lower tile level (tileSize option?), then this is a solved problem)

Mapbox added their optional @2x for raster tiles to dampen this effect, but ideally raster-tiles would be chosen for the specific rendering size (and not necessarily by the tile coverage alone).

However, if you have a small 200x200 pixel iframe and a 512x512 raster tile, then you'd benefit from mipmapping.
This might also be a valid case, so maybe we can even find a good algorithm to smartly turn on/off mipmaps in the future?

Also, there's a visual aspect to this, where having the mips enabled all the time can affect topdown views subtly, making them more blurry in certain zoom levels. I think this is because the GPU starts to sample the second miplevel instead of sticking to the highest miplevel.

Can you show an example? I'd be curious about the tile-sizes / display-sizes where this becomes a problem.
I also wonder if a negative mipmap bias might help.

Also, if this is true, then we might even want to disable mipmap sampling by default until a specified pitch?


Consider me convinced: I think it makes sense to allow users to toggle this, at least until we have a smarter solution.

My thoughts on this topic so far:

@HarelM
Copy link
Member

HarelM commented Dec 6, 2022

Renaming a spec property is not an option due to breaking changes and backwards compatibility which we avoid as much as possible. I don't see a reason to break the style for this edge case feature...
Other than that I would say this should be in the style (I don't have a good name for it) and not both the style and the map options since this will just create confusion.
If this is implemented in the style it should have an issue opened in the native repo for feature parity.
Great discussion and input! Thanks for the thoughts and explanations!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants