feat(sync): add blob engine

[Flrande]

The blob engine completes a part that has been missing in the sync package, from now on all data synchronization of the blocksuite editor (including yjs data and blob data) can be achieved through the sync package.

Breaking changes

Removed the old blob addon (other addons will also be gradually removed and refactored in the future)

Usage of new API

Similar to other engines in the sync package (doc engine and awareness engine), blob engine requires implementing a blob source interface to control the source of blob data. There are two types of data sources: main and shadow, usually one main and multiple shadows, where main is mandatory and shadow is optional. The blob engine will automatically synchronize multiple blob sources, prioritizing data retrieval from the main source.

For example, we implement a blob source based on IndexedDB and use it when initializing DocCollection.

import { createStore, del, get, keys, set } from 'idb-keyval';

class IndexedDBBlobSource implements BlobSource {
  readonly = false;

  readonly store = createStore(`${this.name}_blob`, 'blob');
  readonly mimeTypeStore = createStore(`${this.name}_blob_mime`, 'blob_mime');

  constructor(readonly name: string) {}

  async get(key: string) {
    const res = await get<ArrayBuffer>(key, this.store);
    if (res) {
      return new Blob([res], {
        type: await get(key, this.mimeTypeStore),
      });
    }
    return null;
  }

  async set(key: string, value: Blob) {
    await set(key, value.arrayBuffer(), this.store);
    await set(key, value.type, this.mimeTypeStore);
    return key;
  }

  async delete(key: string) {
    await del(key, this.store);
    await del(key, this.mimeTypeStore);
  }

  async list() {
    const list = await keys<string>(this.store);
    return list;
  }
}

const options: DocCollectionOptions = {
    // ...
    blobSources: {
      main: new IndexedDBBlobSource('quickEdgeless'),
    },
    // ...
  };
const collection = new DocCollection(options);

Another common use case is storing editor's blob data on a remote server. For this purpose, we only need to implement a corresponding blob source. For instance, for testing purposes, we have implemented a mock server blob source.

The sync package has pre-implemented two commonly used blob sources (IndexedDBBlobSource and MemoryBlobSource), which can be used directly.

Close #6294
Close #6752
Close BS-283

Migration Guide

// old
function createBlobStorage(): BlobStorage {
  return {
    crud: {
      set: async (key: string, value: Blob): Promise<string> => {
        await client.insertBlob(key, value);
        return key;
      },
      get: async (key: string): Promise<Blob | null> => {
        return client.getBlob(key);
      },
      delete: async (key: string): Promise<void> => {
        await client.deleteBlob(key);
      },
      list: async (): Promise<string[]> => {
        return client.getAllBlobIds();
      },
    },
  };
}

// new
class ClientBlobSource implements BlobSource {
  readonly = false;
  name = 'client';

  async get(key: string) {
    return client.getBlob(key);
  }

  async set(key: string, value: Blob) {
    return client.insertBlob(key, value);
  }

  async delete(key: string) {
    return client.deleteBlob(key);
  }

  async list() {
    return client.getAllBlobIds();
  }
}

Delete Method

The blob engine does not yet support the deletion of specific blobs, as we currently cannot determine which blobs can be safely deleted, because we cannot confirm whether a blob is used by a doc. For this, we need to wait for the implementation of the indexer.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
blocksuite	❌ Failed (Inspect)			May 20, 2024 10:06am

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
blocksuite-docs	⬜️ Ignored (Inspect)	Visit Preview		May 20, 2024 10:06am

[graphite-app]

Your org has enabled the Graphite merge queue for merging into master

Add the label “merge” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

[doodlewind]

How does this close #6294? I haven't found implementation for uploading image blobs to remote server endpoints yet.

Would also be better to add migration guide, aka the comparison between the new API surface and the implementation based on the legacy BlobStorage. This is needed for bumping downstream (e.g. https://github.com/toeverything/blocksuite/blob/master/examples/react-sqlite/src/editor/provider/provider.ts#L17)

[Flrande]

How does this close #6294? I haven't found implementation for uploading image blobs to remote server endpoints yet.

This depends on the storage service used; perhaps we can write an example based on R2 or S3.

Would also be better to add migration guide, aka the comparison between the new API surface and the implementation based on the legacy BlobStorage. This is needed for bumping downstream (e.g. https://github.com/toeverything/blocksuite/blob/master/examples/react-sqlite/src/editor/provider/provider.ts#L17)

I will update the migration document, and after merging, I will migrate the old implementation in the example.

[fourdim]

LGTM, but please wait for other reviewers.

[Flrande]

Considering the migration cost, I will merge this PR after AFFiNE releases 0.14. cc @pengx17 @EYHN