Feature(python): LanceDB python layer for AWS Lambda

[brnaba-aws]

SDK

Python

Description

Today, the python package for LanceDB is above the maximum package size 250MB. It would be great to have the package optimized to use it as a layer in Lambda. The workaround for now is to dockerized the lambda, which is not ideal.

[wjones127]

I believe this is a duplicate of #1010, no?

[brnaba-aws]

@wjones127 that one if for nodejs right?

[wjones127]

@wjones127 that one if for nodejs right?

Ah we can make this one for Python then. 👍

[wjones127]

Feasibility wise, I think we need to reduce dependency size. IIRC, we require pylance, pyarrow, and numpy. We have a major refactor in progress that will make it possible to have pylance be optional. pyarrow might be harder to eliminate, and IIRC numpy is required by pyarrow (though maybe that has changed).