Support for S3 and PostgreSQL storage backends

[boris-bonin]

Hey everyone,
I wanted to discuss the possibility of implementing support for S3 and PostgreSQL storage backends in the Avalanche HyperSDK as said in https://github.com/ava-labs/hypersdk?tab=readme-ov-file#future-work. Before diving into the development process, I'd like to gather some insights and clarification on what exactly are we looking for from this implementation. Once we have gathered feedback and insights, we can proceed with defining the scope of work and creating a development roadmap.

Key Points for Discussion:

• Use Cases: What are the primary use cases for incorporating S3 and PostgreSQL storage backends?
• Functionality: What specific functionalities should be supported by the implementation?
• Security: What security measures should be implemented to ensure the safety and integrity of data stored in S3 and PostgreSQL?

I look forward to hearing your thoughts and suggestions on this topic!

[richardpringle]

Given some of my experience dealing with nodes/clients that require a particular backend that isn't embedded, it would be ideal if, by default, there was a local backend. I should be able to spin something up with minimal configuration. When messing around with a deployment.

We can print a warning "for development purposes only" or something if an external storage backend isn't specified.

[patrick-ogrady]

I plan to implement some of this flow with: #961

I think of it more as a reliable pipeline out of the node to some external store/sidecar. The node itself wouldn't use either datastore to service anything (at least initially).

In terms of integrity, I think blocks/transactions would be straightforward (anything retrieved must match the hash). State integrity is much more complex (unless some proof mechanism was also built on top). Its unclear if that is required initially (verified reads can rely on sending a message to the chain).

[kpachhai]

Interested in this so checking the status of this.