Async Actor and version 4.x first breaking changes (#3233)

* Initial concept * More improvements * Fix lint * Main commit to move most of the actor code to use promises and register. There are still things to do. * Fix actor tests * Remove more any when it comes to worker self typings. * Improve more types, add more tests * Improve more types, add more tests * Fix lint * Fix more tests * Fix tests * Fix some sytle tests * Add comment for todo. * Fix test related to message parameters change * Fix remaining tests * Added todo to accomodate for code review. * Fix build test * Fix geojson types * Rename some events, remove clutter a bit * Fix issue with multiple maps found in integration tests * Fix tests and improve types and docs a bit more * Final fixes to make the tests pass * Fix build test * a couple of tweaks (#3267) * Add changelog item * More improvement to async `Actor` and replace more callbacks with promises (#3269) * Improve vector tile source * Fix unit tests * Fix render tests for the case of 404 * Fix lint * Remove unused method * More improvements to remove callbacks and move to promises * Fixed some tests, more fixes are needed. * Fix remaining tests. * Fix lint * Fix tile parser code to pass render tests * Fix lint * Fix lint * More fixes and tests in the infrasturcure when it comes to cancelling * forgot to inclue this... * Fix lint and build * Convert parse to be a proper async method. * Add more test coverate to actor, allow more testing in mock web worker. * Replace getJson call with async code and remove more callbacks (#3273) * Add abort between actors of main thread and worker thread with tests * Fix lint * Fix lint * Fix node from lts to nvmrc * Improve the code of vector tile and geojson workers * Fix lint * Add debug when tests are failing * Do not throw in case of empty geojson tile * Fix lint * Move geojson worker source to be more async like. * Change getJSON to be async extenally * Update src/ui/map.test.ts Co-authored-by: Bart Louwers <[email protected]> --------- Co-authored-by: Bart Louwers <[email protected]> * Move image queue to use promises, move getArrayBuffer to use promises (#3280) * Migrate getArrayBuffer to be promise based * Remove dead code in ajax.ts file * Code review changes (#3278) Co-authored-by: neodescis <[email protected]> * Make image request return promises. * Move process queue to the right location * Increase build size * Last fixes * fix lint warning * Fix lint * update docs Co-authored-by: neodescis <[email protected]> * Update docs Co-authored-by: neodescis <[email protected]> * Fix docs * Code review comments. * Update src/util/image_request.ts * Update src/util/util.ts --------- Co-authored-by: neodescis <[email protected]> Co-authored-by: neodescis <[email protected]> * Remove TODOs, tidy up last things (#3301) * Remove some TODOs * Remove some todos, make ajax API fully promise based * Move tileJson to be async * fix lint * Remove another TODO * Move makexmlhttprequest to return a promise * Make load_sprite async * Added relevant tests * Final fixes * Update src/source/vector_tile_source.test.ts Co-authored-by: neodescis <[email protected]> --------- Co-authored-by: neodescis <[email protected]> * Async actor docs additions (#3305) * Fix missing classes in the docs * Fixed the relevant docs in the places where API calls were changed. * Fix lint * Remove some "then" from tests * remove todos, remove "then"s, simplify the ajax code a bit * Remove as any from where it's not needed * Fix code review expect stuff... * explicit promises instead of array. * Improve ajax further * Improve docs for RTL plugin status and reduce code in geojson worker test * Last simplification to actor, simplify a style test. * Add missing docs comments * Fix lint... * Fix Actor XSS, introduce subscribe (#3329) * Fix XSS, intorduce subscribe * Update changelog * Fix developer diagram with updated flow * Async actor no log warnings and errors in unit tests (#3368) * Upgrade jest, fix tests, remove console log messages. * Fix incorrect additions, lint * Async actor remove cancelables (#3371) * Remove more places that use cancelable * Remove more places with canceable * Async actor callback and promise (#3374) * Remove more places that use cancelable * Remove more places with canceable * callback and promise * Improve sources code when it is related to callback * Use callback failback when needed * Update build size test * Removed callbacks from various places, adding docs, simplifying tests * Improve type * Update src/util/actor.test.ts Co-authored-by: neodescis <[email protected]> * Make load async for all sources --------- Co-authored-by: neodescis <[email protected]> * Update changelog with most of the changes --------- Co-authored-by: Michael Barry <[email protected]> Co-authored-by: Bart Louwers <[email protected]> Co-authored-by: neodescis <[email protected]> Co-authored-by: neodescis <[email protected]>
maplibre · Nov 23, 2023 · 859b1a7 · 859b1a7
1 parent 71f092a
commit 859b1a7
Show file tree

Hide file tree

Showing 89 changed files with 3,550 additions and 3,768 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,9 +1,17 @@
 ## main
 
 ### ✨ Features and improvements
+
+- Changed `ImageRequest` to be `Promise` based ([#3233](https://github.com/maplibre/maplibre-gl-js/pull/3233))
+- ⚠️ Changed the undeling worker communication from callbacks to promises. This has a breaking effect on the implementation of custom `WorkerSource` and how it behaves ([#3233](https://github.com/maplibre/maplibre-gl-js/pull/3233))
+- ⚠️ Changed the `Source` interface to return promises instead of callbacks ([#3233](https://github.com/maplibre/maplibre-gl-js/pull/3233))
+- ⚠️ Changed all the sources to be promises based. ([#3233](https://github.com/maplibre/maplibre-gl-js/pull/3233))
+- ⚠️ Changed the `map.loadImage` method to return a `Promise` ([#3233](https://github.com/maplibre/maplibre-gl-js/pull/3233))
 - _...Add new stuff here..._
 
 ### 🐞 Bug fixes
+
+- Fixes a security issue in `Actor` against XSS attacks in postMessage / onmessage ([#3239](https://github.com/maplibre/maplibre-gl-js/pull/3239))
 - _...Add new stuff here..._
 
 ## 3.6.2

diff --git a/build/generate-doc-images.ts b/build/generate-doc-images.ts
@@ -2,6 +2,7 @@ import path from 'path';
 import fs from 'fs';
 import puppeteer from 'puppeteer';
 import packageJson from '../package.json' assert { type: 'json' };
+import {sleep} from '../src/util/test/util';
 
 const exampleName = process.argv[2];
 const examplePath = path.resolve('test', 'examples');
@@ -28,10 +29,8 @@ async function createImage(exampleName) {
         .then(async () => {
             // Wait for 5 seconds on 3d model examples, since this takes longer to load.
             const waitTime = exampleName.includes('3d-model') ? 5000 : 1500;
-            await new Promise((resolve) => {
-                console.log(`waiting for ${waitTime} ms`);
-                setTimeout(resolve, waitTime);
-            });
+            console.log(`waiting for ${waitTime} ms`);
+            await sleep(waitTime);
         })
         // map.loaded() does not evaluate to true within 3 seconds, it's probably an animated example.
         // In this case we take the screenshot immediately.

diff --git a/developer-guides/life-of-a-tile.md b/developer-guides/life-of-a-tile.md
@@ -93,7 +93,7 @@ sequenceDiagram
         worker_tile->glyph manager: getImages
         glyph manager->>ajax: Fetch icon<br>images
         glyph manager-->>worker_tile: glyph/Image dependencies
-        worker_tile->>worker_tile: maybePrepare()
+        worker_tile->>worker_tile: wait for all requests to finish
         worker_tile->>worker_tile: create GlyphAtlas
         worker_tile->>worker_tile: create ImageAtlas
         worker_tile->>bucket: addFeatures
@@ -169,19 +169,19 @@ sequenceDiagram
     map->>painter: render(style)
     painter->>source_cache: prepare(context)
     loop for each tile
-    source_cache->>GPU: upload vertices
-    source_cache->>GPU: upload image textures
+        source_cache->>GPU: upload vertices
+        source_cache->>GPU: upload image textures
     end
     loop for each layer
-    painter->>layer: renderLayer(pass=offscreen)
-    painter->>layer: renderLayer(pass=opaque)
-    painter->>layer: renderLayer(pass=translucent)
-    painter->>layer: renderLayer(pass=debug)
-    loop renderLayer() call for each tile
-    layer->>GPU: load program
-    layer->>GPU: drawElements()
-    GPU->>user: display pixels
-    end
+        painter->>layer: renderLayer(pass=offscreen)
+        painter->>layer: renderLayer(pass=opaque)
+        painter->>layer: renderLayer(pass=translucent)
+        painter->>layer: renderLayer(pass=debug)
+        loop renderLayer() call for each tile
+            layer->>GPU: load program
+            layer->>GPU: drawElements()
+            GPU->>user: display pixels
+        end
     end
     map->>map: triggerRepaint()
 ```

diff --git a/src/data/dem_data.ts b/src/data/dem_data.ts
@@ -3,20 +3,24 @@ import {RGBAImage} from '../util/image';
 import {warnOnce} from '../util/util';
 import {register} from '../util/web_worker_transfer';
 
-// DEMData is a data structure for decoding, backfilling, and storing elevation data for processing in the hillshade shaders
-// data can be populated either from a pngraw image tile or from serliazed data sent back from a worker. When data is initially
-// loaded from a image tile, we decode the pixel values using the appropriate decoding formula, but we store the
-// elevation data as an Int32 value. we add 65536 (2^16) to eliminate negative values and enable the use of
-// integer overflow when creating the texture used in the hillshadePrepare step.
-
-// DEMData also handles the backfilling of data from a tile's neighboring tiles. This is necessary because we use a pixel's 8
-// surrounding pixel values to compute the slope at that pixel, and we cannot accurately calculate the slope at pixels on a
-// tile's edge without backfilling from neighboring tiles.
-
+/**
+ * The possible DEM encoding types
+ */
 export type DEMEncoding = 'mapbox' | 'terrarium' | 'custom'
 
+/**
+ * DEMData is a data structure for decoding, backfilling, and storing elevation data for processing in the hillshade shaders
+ * data can be populated either from a pngraw image tile or from serliazed data sent back from a worker. When data is initially
+ * loaded from a image tile, we decode the pixel values using the appropriate decoding formula, but we store the
+ * elevation data as an Int32 value. we add 65536 (2^16) to eliminate negative values and enable the use of
+ * integer overflow when creating the texture used in the hillshadePrepare step.
+ *
+ * DEMData also handles the backfilling of data from a tile's neighboring tiles. This is necessary because we use a pixel's 8
+ * surrounding pixel values to compute the slope at that pixel, and we cannot accurately calculate the slope at pixels on a
+ * tile's edge without backfilling from neighboring tiles.
+ */
 export class DEMData {
-    uid: string;
+    uid: string | number;
     data: Uint32Array;
     stride: number;
     dim: number;
@@ -27,9 +31,18 @@ export class DEMData {
     blueFactor: number;
     baseShift: number;
 
-    // RGBAImage data has uniform 1px padding on all sides: square tile edge size defines stride
+    /**
+     * Constructs a `DEMData` object
+     * @param uid - the tile's unique id
+     * @param data - RGBAImage data has uniform 1px padding on all sides: square tile edge size defines stride
     // and dim is calculated as stride - 2.
-    constructor(uid: string, data: RGBAImage, encoding: DEMEncoding, redFactor = 1.0, greenFactor = 1.0, blueFactor = 1.0, baseShift = 0.0) {
+     * @param encoding - the encoding type of the data
+     * @param redFactor - the red channel factor used to unpack the data, used for `custom` encoding only
+     * @param greenFactor - the green channel factor used to unpack the data, used for `custom` encoding only
+     * @param blueFactor - the blue channel factor used to unpack the data, used for `custom` encoding only
+     * @param baseShift - the base shift used to unpack the data, used for `custom` encoding only
+     */
+    constructor(uid: string | number, data: RGBAImage | ImageData, encoding: DEMEncoding, redFactor = 1.0, greenFactor = 1.0, blueFactor = 1.0, baseShift = 0.0) {
         this.uid = uid;
         if (data.height !== data.width) throw new RangeError('DEM tiles must be square');
         if (encoding && !['mapbox', 'terrarium', 'custom'].includes(encoding)) {

diff --git a/src/data/feature_index.ts b/src/data/feature_index.ts
@@ -40,7 +40,6 @@ type QueryParameters = {
 };
 
 /**
- * @internal
  * An in memory index class to allow fast interaction with features
  */
 export class FeatureIndex {

diff --git a/src/index.test.ts b/src/index.test.ts
@@ -2,6 +2,7 @@ import {config} from './util/config';
 import maplibre from './index';
 import {getJSON, getArrayBuffer} from './util/ajax';
 import {ImageRequest} from './util/image_request';
+import {isAbortError} from './util/abort_error';
 
 describe('maplibre', () => {
     beforeEach(() => {
@@ -34,65 +35,55 @@ describe('maplibre', () => {
         expect(Object.keys(config.REGISTERED_PROTOCOLS)).toHaveLength(0);
     });
 
-    test('#addProtocol - getJSON', done => {
+    test('#addProtocol - getJSON', async () => {
         let protocolCallbackCalled = false;
         maplibre.addProtocol('custom', (reqParam, callback) => {
             protocolCallbackCalled = true;
             callback(null, {'foo': 'bar'});
             return {cancel: () => {}};
         });
-        getJSON({url: 'custom://test/url/json'}, (error, data) => {
-            expect(error).toBeFalsy();
-            expect(data).toEqual({foo: 'bar'});
-            expect(protocolCallbackCalled).toBeTruthy();
-            done();
-        });
+        const response = await getJSON({url: 'custom://test/url/json'}, new AbortController());
+        expect(response.data).toEqual({foo: 'bar'});
+        expect(protocolCallbackCalled).toBeTruthy();
     });
 
-    test('#addProtocol - getArrayBuffer', done => {
+    test('#addProtocol - getArrayBuffer', async () => {
         let protocolCallbackCalled = false;
-        maplibre.addProtocol('custom', (reqParam, callback) => {
+        maplibre.addProtocol('custom', (_reqParam, callback) => {
             protocolCallbackCalled = true;
-            callback(null, new ArrayBuffer(1));
+            callback(null, new ArrayBuffer(1), 'cache-control', 'expires');
             return {cancel: () => {}};
         });
-        getArrayBuffer({url: 'custom://test/url/getArrayBuffer'}, async (error, data) => {
-            expect(error).toBeFalsy();
-            expect(data).toBeInstanceOf(ArrayBuffer);
-            expect(protocolCallbackCalled).toBeTruthy();
-            done();
-        });
+        const response = await getArrayBuffer({url: 'custom://test/url/getArrayBuffer'}, new AbortController());
+        expect(response.data).toBeInstanceOf(ArrayBuffer);
+        expect(response.cacheControl).toBe('cache-control');
+        expect(response.expires).toBe('expires');
+        expect(protocolCallbackCalled).toBeTruthy();
     });
 
-    test('#addProtocol - returning ImageBitmap for getImage', done => {
+    test('#addProtocol - returning ImageBitmap for getImage', async () => {
         let protocolCallbackCalled = false;
-        maplibre.addProtocol('custom', (reqParam, callback) => {
+        maplibre.addProtocol('custom', (_reqParam, callback) => {
             protocolCallbackCalled = true;
             callback(null, new ImageBitmap());
             return {cancel: () => {}};
         });
 
-        ImageRequest.getImage({url: 'custom://test/url/getImage'}, async (error, img) => {
-            expect(error).toBeFalsy();
-            expect(img).toBeInstanceOf(ImageBitmap);
-            expect(protocolCallbackCalled).toBeTruthy();
-            done();
-        });
+        const img = await ImageRequest.getImage({url: 'custom://test/url/getImage'}, new AbortController());
+        expect(img.data).toBeInstanceOf(ImageBitmap);
+        expect(protocolCallbackCalled).toBeTruthy();
     });
 
-    test('#addProtocol - returning HTMLImageElement for getImage', done => {
+    test('#addProtocol - returning HTMLImageElement for getImage', async () => {
         let protocolCallbackCalled = false;
         maplibre.addProtocol('custom', (reqParam, callback) => {
             protocolCallbackCalled = true;
             callback(null, new Image());
             return {cancel: () => {}};
         });
-        ImageRequest.getImage({url: 'custom://test/url/getImage'}, async (error, img) => {
-            expect(error).toBeFalsy();
-            expect(img).toBeInstanceOf(HTMLImageElement);
-            expect(protocolCallbackCalled).toBeTruthy();
-            done();
-        });
+        const img = await ImageRequest.getImage({url: 'custom://test/url/getImage'}, new AbortController());
+        expect(img.data).toBeInstanceOf(HTMLImageElement);
+        expect(protocolCallbackCalled).toBeTruthy();
     });
 
     test('#addProtocol - error', () => {
@@ -101,20 +92,27 @@ describe('maplibre', () => {
             return {cancel: () => { }};
         });
 
-        getJSON({url: 'custom://test/url/json'}, (error) => {
+        getJSON({url: 'custom://test/url/json'}, new AbortController()).catch((error) => {
             expect(error).toBeTruthy();
         });
     });
 
-    test('#addProtocol - Cancel request', () => {
+    test('#addProtocol - Cancel request', async () => {
         let cancelCalled = false;
         maplibre.addProtocol('custom', () => {
             return {cancel: () => {
                 cancelCalled = true;
             }};
         });
-        const request = getJSON({url: 'custom://test/url/json'}, () => { });
-        request.cancel();
+        const abortController = new AbortController();
+        const promise = getJSON({url: 'custom://test/url/json'}, abortController);
+        abortController.abort();
+        try {
+            await promise;
+        } catch (err) {
+            expect(isAbortError(err)).toBeTruthy();
+        }
+
         expect(cancelCalled).toBeTruthy();
     });
 

diff --git a/src/render/glyph_atlas.ts b/src/render/glyph_atlas.ts
@@ -2,7 +2,8 @@ import {AlphaImage} from '../util/image';
 import {register} from '../util/web_worker_transfer';
 import potpack from 'potpack';
 
-import type {GlyphMetrics, StyleGlyph} from '../style/style_glyph';
+import type {GlyphMetrics} from '../style/style_glyph';
+import type {GetGlyphsResponse} from '../util/actor_messages';
 
 const padding = 1;
 
@@ -37,11 +38,7 @@ export class GlyphAtlas {
     image: AlphaImage;
     positions: GlyphPositions;
 
-    constructor(stacks: {
-        [_: string]: {
-            [_: number]: StyleGlyph;
-        };
-    }) {
+    constructor(stacks: GetGlyphsResponse) {
         const positions = {};
         const bins = [];
-Original file line number
+Diff line change
@@ Expand Up / @@ -40,7 +40,6 @@ type QueryParameters = { @@
     };
     /**
-     * @internal
      * An in memory index class to allow fast interaction with features
      */
     export class FeatureIndex {
@@ Expand Down @@