How to backfill GSI for new entity index?

[pablote]

Imagine this situation:

1. You create a table with GSI 1 already there.
2. Add a new entity with only local pk and sk
3. Rows are inserted for the new entity
4. Come up with new access pattern and index for said entity, map it GSI 1

Only newly inserted rows will be present in the index, see the following screenshot:

I imagine I can write a script that hits DynamoDB directly and fills all the missing gsi1pk and gsi1sk, but is there a way to do this from electrodb itself? I'd much rather that'd be case.

It doesn't have to be something automatic at startup, 'cause I imagine there'd be a cost to this. But if I know I'm in the situation, it'd be great to do something like: MyEntity.byMyCondition.backfill().

[Mikbol]

I'm also looking for a better way to handle this.
This must be a common scenario during development?

Is there a better solution around than having a one-time job that fetches all entities and then updates/put them immediately?

// without handling errors, batching, unprocessed entities, etc.
const allMyEntities = // get all entities by looping queries, scan, etc.
await MyEntity.put(allMyEntities).go({ pages: "all" })

[NicoPowers]

I think having a separate repo that could resemble electrodb migration tools or some kind of cli designed to work with electrodb would good. This would need to take the different in schema versions (by checking the entity version) and then generate the commands needed to backfill version 1 with 2, etc.

[zirkelc]

It would be nice to have a built-in function which you can call on entity-level. Here's a generic but ineffecient workaround:

type BackfillIndexOptions = {
  tableName?: string;
  region?: string;
};
export const backfillIndex = async <E extends Entity<any, any, any, any>>(
  entity: E,
  options: BackfillIndexOptions = {},
): Promise<void> => {
  if (options.tableName) {
    entity.setTableName(options.tableName);
  }

  if (options.region) {
    entity.setClient(new DynamoDBClient({ region: options.region }));
  }

  const items = await entity.scan.go({ pages: 'all' });

  for (const item of items.data) {
    const result = await entity
      .put(item)
      .where(({ updatedAt }, { eq }) => eq(updatedAt, item.updatedAt))
      .go();

    console.log('Updated item', result);
  }
}

[tywalch]

Migrations end up being very specific to the details of the project, and often have unique constraints/considerations (e.g. additional operations/calls per item, rate considerations, concurrency optimizations, custom logic to handle mid-migration failure, etc).

@zirkelc your example may work best for smaller tables -- scan with pages: 'all' will pull all items into memory before writing. For larger tables, it might be better to perform a put on each individual page from the scan.

How you model your entity also has a huge impact on how you can performantly migrate the data. If I'm migrating a "users" table, I might first get "organizations" first which then allows me to get all users for each organization in parallel.

A lot of ways to skin the cat here, which is why there isn't a built in tool 😅