Roadmap for future releases

[mesozoic]

The following is a rough proposal for which features we plan to add to pyAirtable, loosely grouped into releases. Priorities can shift over time, and anyone who wants to see a particular feature implemented faster is welcome to submit pull requests for review. I'm comfortable with the scope of the 1.5.0 and 2.0.0 releases; everything past there is a bit more speculative, but I have proofs of concept for at least some of it.

This document assumes we are using semantic versioning, and that existing APIs can only be broken if there is a major version number change. As such, we may be more cautious about implementing new endpoints (or accepting PRs for new endpoints) if the design of that API merits discussion, because we'll commit to supporting new APIs for at least one major version.

Tagging other recent code contributors for input. @BAPCon @goksan @larsakerson @NicoHood

1.5 (released May '23)

This release will focus on bug fixes, quick wins, and reducing the open issue count.

• Documentation typos and fixes (see 197)
• Implement continuous integration with type checking (see 195, 217, 239, 240, 241, 242)
• configurable api_base_url #243
• Add ORM LookupField #182
• Ignore extra fields in ORM (fixes #190, option 1) #250 (see 190)
• Use POST /listRecords instead of GET when the URL gets too long #247 (see 203, 210, 222)
• {Api,Base,Table}.batch_upsert and docs #255 (see 209, 216)
• Create and update commands do not take return_fields_by_field_id into account #194

2.0 (July '23)

This release will be our chance to make any breaking API changes, such as changes to default behavior or return types. We will also implement stricter typing across the library, which might break CI for downstream code that does its own type checking, and support every field type in the ORM.

• Remove references/support for Python 3.7 #261
• Support readonly fields and all Airtable field types in ORM #260
• Return models, not dicts, from Model.{all,first} #262
• Remove (or fully implement) validate_type= parameter to Field constructor
• Support mypy --strict #263 (see 202)
• Use dataclasses or TypedDict on all return values
• Support urllib3 >= 2 #251
• Drop ApiAbstract #267
• Use a default retry strategy and remove time.sleep #272
• Implement api.whoami() #273 for Get user ID & scopes
• batch_save and batch_delete for ORM models #274
• Move ORM out of "experimental" status #277
• Return full payload from batch_upsert #281
• CRUD support for comments #282
• Allow ORM Meta options to be lookups at runtime (not import time) #268
• ORM: Support passing LinkField by name (see 176, 200)

2.1

Just a plain ol' feature release.

• Webhooks (see 208, 236)
  • List webhook payloads
  • List webhooks
  • Create a webhook
  • Delete a webhook
  • Enable/disable webhook notifications
  • Refresh a webhook

2.2

Build a sustainable pattern for retrieving with Airtable metadata, which involves a lot of deeply nested object structures.

• Enterprise: Read enterprise information
  • Get enterprise
  • Get user by email
  • Get user by id
  • Get user group
• Metadata APIs that return typed dataclasses instead of dicts
• Metadata: Read base, table, field schemas (see 204, 230, 231)
  • List bases
  • Get base schema
  • List views
  • Get view metadata
  • Get base collaborators
  • Get interface
• Metadata: Read collaborations/sharing
  • Get workspace collaborators
  • List shares
• Metadata: Modify base, table, field schemas (see 235, 237)
  • Update field
  • Create field
  • Update table
  • Create table
  • Create base
  • Delete view
  • Delete base
  • Delete workspace
  • Move base

2.3

• Enterprise: Audit log events
• Start using trusted publishing for PyPI
• Metadata: Modify collaborations/sharing
  • Update collaborator base permission
  • Update interface collaborator
  • Update workspace collaborator
  • Delete base collaborator
  • Delete interface collaborator
  • Delete workspace collaborator
  • Add base collaborator
  • Add interface collaborator
  • Add workspace collaborator
  • Delete base invite
  • Delete interface invite
  • Delete workspace invite
  • Delete share
  • Manage share
  • Update workspace restrictions
• Enterprise: Manage users
  • Remove user from enterprise
  • Delete user by id
  • Delete users by email
  • Logout user
  • Manage user
  • Manage user batched
  • Manage user membership

3.0

Next major version bump after 2.0 can be for bigger ideas.

• Revisit the DSL for building Airtable formulas.
• Testing utilities for library users to safely mock the pyAirtable API in their own tests.
• Command-line interface for retrieving (or modifying?) Airtable data and metadata.
• Auto-generate ORM models from Airtable base metadata.

4.0

• Drop support for Python 3.8 (EOL Oct '24)

Won't do

The following wound up being a lot more work than we'd anticipated.

• Run doctests in pyairtable and docs/source on build.

The following seem like uncommon use cases and we'll wait for another contributor to need them.

• Metadata: Manage block installations
  • List block installations
  • Delete block installation
  • Manage block installation
• Sync CSV data
  • Sync CSV data

SCIM is a standard API specification that is already implemented by any vendors which need to support it, so implementing it in pyAirtable doesn't seem like a priority.

• SCIM: Manage groups/users
  • Create group
  • Create user
  • Delete group
  • Patch group
  • Patch user
  • Put group
  • Put user
• SCIM: Retrieve information
  • List groups
  • List users
  • Get group
  • Get user

[gtalarico]

Thanks for putting this together @mesozoic; this looks like a great plan.

Would 2.x be a good time to also move ORM package out of experimental and make it officially supported?

Auto-generate ORM models from Airtable base metadata
😍

[mesozoic]

Would 2.x be a good time to also move ORM package out of experimental and make it officially supported?

Totally! Though guessing from the comments, it seems like plenty of folks are using it already, supported or otherwise 😁 so I'm still inclined to do a major version bump before changing the semantics, return types, etc.

[walkingpendulum]

I am so hyped for test-related features in 3.0.0 😍

[mesozoic]

I am so hyped for test-related features in 3.0.0 😍

Great! Can you share a bit more about your use case (or existing solutions) to help inform what direction we take it?

[walkingpendulum]

we have dozen of tables, couple of them having up to 200 fields, most of which are lookups, rollups and formulas.

major issues with Airtable development, exploitation and testing:

• no table-schema-as-a-code. people mess up with field names in UI and it break our python scripts and JS automations. things get worse when link field is renamed, it breaks all the downstream lookups and rollups. we are also continuously discovering and fixing discrepancies between table schemes jn prod and dev tables
• generally we prefer stubs to mocks in tests. for example, we spin MongoDB, RabbitMQ, and Moto (as a drop-in replacement for s3) containers in tests. this approach is not working for pyairtable, there is no stub for Airtable rest api available. which is really inconvenient, because our typical unit of logic deals with 5-10 fields, most of which are lookups and rollups (this is why I am also happy for close ORM features release), and it is a nightmare to mock, test and assert
• ORM mode (soon to be released) will cone very handy. as I mentioned, there are tons of lookups and rollups, which lead to spaghetti code without ORM mode

[mesozoic]

@walkingpendulum Your use case sounds pretty complicated! So far we've avoided the field name issues by locking down Creator privileges for the bases where we've built a lot of heavy integrations. Sounds like that might not be feasible for you.

We haven't really come up with a great solution to testing; we have a minimal "fake Airtable" that mocks out the API's basic behaviors in-memory, and for integration testing we have a "partitioning shim" that injects some extra formulas/fields into every API call so that multiple test runners can operate on a single base without interfering w/ each other (as long as there's a text field where they can store a unique ID per test run). Both are utilities we'd like to move from our repos into upstream.

[walkingpendulum]

hello, guys! it's been a long time :)

I'd like to touch base on the 3.0 feature "Testing utilities for library users to safely mock the pyAirtable API in their own tests". do you still aim for it? I am as hyped to see it as I was a year ago.

also - great job done! we have moved from one-fits-all connector component that isolates AT API calls to ad-hoc pyairtable calls, because we realized how simple and powerful pyairtable is, and never looked back. thank you for your work.

[mesozoic]

@walkingpendulum I'd still like to move what we have upstream at some point, but it's not a high priority at the moment. I'm glad some of the other improvements to the library have worked out for you!

[acmeguy]

Is it possible for Table.all() functions to return the label/value of a Lookup field as well as the record id?
(This is all dynamic and I can not use the ORM)

[d717an]

Is it possible to enable passing parameters to the underlying request implementation when creating the Api(token) instance? For example, I need to pass in a path to a pem file. With the requests library, I do this with a verify=path_to_pem parameter.