can't specify attributes in scan operation?

[anatolzak]

Describe the bug
When performing a get or query operation, you can pass attributes to the go or params method.
However when performing a scan operation, you cannot pass attributes. is this by design?

It would be nice to be able to specify attributes when performing a scan to not pull unnecessary data over the wire.

ElectroDB Version
2.15.0

[anatolzak]

I was able to come up with a hacky workaround that allows specifying the attributes for a scan operation with fully type-safe results. Here is an example of the API

const Product = new Entity({
 ...
 attributes: {
    userId: { type: 'string', required: true },
    productId: { type: 'string', required: true },
    title: { type: 'string', required: true },
    price: { type: 'number', required: true },
    imageUrl: { type: 'string', required: true },
  },
 ...
});

const query = Product.scan.where(({ price }, { gt }) => gt(price, 100));

const options = getScanOptionsWithProjection({
  entity: Product,
  attributes: ['productId', 'title'], // type safe
  getParams: query.params,
  options: {
    // any other options (e.g `order`, etc.)
  },
});

const { data }: ScanResultWithProjection<typeof options> = await query.go(options);
//       ^? { productId: string; title: string; }[]

This produces the following dynamo command:

{
    "TableName": "products",
    "ExpressionAttributeNames": {
        "#price": "price",
        "#__edb_e__": "__edb_e__",
        "#__edb_v__": "__edb_v__",
        "#pk": "pk",
        "#sk": "sk",
        "#productId": "productId",
        "#title": "title"
    },
    "ExpressionAttributeValues": {
        ":price0": 100,
        ":__edb_e__0": "product",
        ":__edb_v__0": "1",
        ":pk": "$app#userid_",
        ":sk": "$product_1#productid_"
    },
    "FilterExpression": "begins_with(#pk, :pk) AND begins_with(#sk, :sk) AND (#price > :price0) AND #__edb_e__ = :__edb_e__0 AND #__edb_v__ = :__edb_v__0",
    "ProjectionExpression": "#__edb_e__, #__edb_v__, #productId, #title"
}

And here is the code that make this happen:

declare const __brand: unique symbol;
type Brand<T, U> = T & { [__brand]: U };

export type EntityAttributeKey<T> =
  T extends Entity<any, any, any, any>
  ? keyof T['schema']['attributes']
  : never;

export type ScanResultWithProjection<
  TOptions extends {
    [__brand]: {
      entity: Entity<any, any, any, any>;
      attributes: EntityAttributeKey<TOptions[typeof __brand]['entity']>[];
    };
  },
  > = {
    data: Pick<
      EntityItem<TOptions[typeof __brand]['entity']>,
      TOptions[typeof __brand]['attributes'][number]
    >[];
    cursor: string | null;
  };

export function getScanOptionsWithProjection<
  TEntity extends Entity<any, any, any, any>,
  TAttributes extends EntityAttributeKey<TEntity>[],
  TOptions extends QueryOptions = {},
  >({
    attributes,
    getParams,
    options = {} as TOptions,
  }: {
    entity: TEntity;
    attributes: TAttributes;
    getParams: (opts: QueryOptions) => object;
    options?: TOptions;
  }) {
  const attributesSet = new Set<string>([
    '__edb_e__',
    '__edb_v__',
    ...(attributes as any),
  ]);

  const params = getParams(options);

  const { ExpressionAttributeNames } = params as {
    ExpressionAttributeNames: Record<string, string>;
  };

  for (const attr of attributesSet) {
    ExpressionAttributeNames[`#${attr}`] = attr;
  }

  const mergedOptions = {
    ...options,
    params: {
      ...params,
      ExpressionAttributeNames,
      ProjectionExpression: Array.from(attributesSet)
        .map((attr) => `#${attr}`)
        .join(', '),
    },
  } as const;

  return mergedOptions as Brand<
    typeof mergedOptions,
    {
      entity: TEntity;
      attributes: TAttributes;
    }
  >;
}

Link to ElectroDB playground

[tywalch]

Great request! I'm surprised it's not an option currently tbh

[anatolzak]

@tywalch just wanted to express my deep appreciation for what you have done with ElectroDB!

ElectroDB makes it a breeze to work with Dynamo and covers 99% of cases. And thanks to the escape hatches you have implemented, I can implement functionality that ElectroDB doesn't support relatively easily.

One example is my workaround for specifying the attributes in a scan operation. Another example is the ability to scan a GSI which is not natively supported by ElectroDB but can be easily accomplished by performing a scan and passing the index name in go({ params: {...} }) like this

const userCheapProducts = await Product.query.byUserAndCheap({ userId: '123' }).go();

const allCheapProducts = await Product.scan.go({
  params: {
    IndexName: Product.schema.indexes.byUserAndCheap.index,
  },
});

const Product = new Entity({
  attributes: {
    userId: { type: 'string', required: true },
    productId: { type: 'string', required: true },
    price: { type: 'number', required: true },
  },
  indexes: {
    byUser: {
      pk: {
        field: 'pk',
        composite: ['userId'],
      },
      sk: {
        field: 'sk',
        composite: ['productId'],
      },
    },
    byUserAndCheap: {
      index: 'gsi1',
      condition: (attrs) => {
        // checking for undefined prevents the index from
        // being deleted during updates that omit the price field
        return attrs.price === undefined || attrs.price < 100;
      },
      pk: {
        field: 'gsi1pk',
        composite: ['userId'],
      },
      sk: {
        field: 'gsi1sk',
        composite: ['price', 'productId'],
      },
    },
  },
});

again, thanks so much! 🙏