How should site handle business logic?

[armincerf]

Currently, Site is focused only on building custom APIs for XTDB, but you are limited to providing datalog queries. Any processing that can't be done in a datalog query must be done on the frontend currently.

So the questions to be answered are:

• Where does custom logic run? (on the site server? on a service such as AWS Lambda?)
• How is the custom logic defined (both the API details such as endpoint/parameters and the code itself)
• Where does the custom logic live (is it put into XTDB or does site have no opinions about this)
• How are event triggers (e.g send an email after a user is added) done, presumable need some queue
• How are the results from running custom logic used in an XT query (e.g fetch from some external source and query XT based on result)
• How are the results of running an XT query passed to some custom logic (e.g query something from XT and use it as a parameter in a custom function)

[armincerf]

Some resources gathered so far (will post separate answers to make comment threads easier):

• Event queue system, could implement a version for site to allow for event triggers to be attached to queries https://github.com/ivarref/yoltq

[reborg]

This is a Datomic-backed event queue suitable for implementing the transactional outbox microservice pattern. It is essentially a replacement for two phase commit when this is not an option. The typical email use case would work as follow: the domain update and the sending email action would be in the same transaction. The send email action is just about storing the action on to a persistent "table" (the queue). Another service picks up the sending email action asynchronously. If for any reason the domain update rolls back, the email action is not persisted to the queue. I guess it would be nice fit for XT as well.

[armincerf]

Architecture proposal which seems quite similar to Site's aims https://3factor.app/ - relevant HN thread https://news.ycombinator.com/item?id=18255574

[armincerf]

hasura's event trigger system is their solution to #2, you subscribe to the event query and the client is streamed updates to that event

[reborg]

hasura's event trigger system is their solution to #2, you subscribe to the event query and the client is streamed updates to that event

I see. I still find it a bit confusing. The diagram in the 3Factor site seems to imply that the database disappears going from old to new. In Hasura the database triggers the events and should be part of the picture. Hasura seems to implement a scenario where you want the legacy DB to stay where it is.

[armincerf]

Hmm what do you mean by legacy db? hasura has nothing to do with legacy DBs (though you can connect a legacy db to it if you want to, it doesn't seem to be the main use case)

In this diagram

The graphql API, state and event system are all managed by hasura (though the state could be your own postgres instance that you connect to hasura)

The serverless functions are not managed by hasura, you just provide the webhook URLs.

[reborg]

Hmm what do you mean by legacy db? hasura has nothing to do with legacy DBs (though you can connect a legacy db to it if you want to, it doesn't seem to be the main use case)

Perhaps I'm making it too specific to a use case where you want to kill a monolithic database. I meant "legacy" to indicate whatever they call "your database" on the main page. But sure, it could be that you're perfectly happy with your database and you want events generated from it. Since Hasura requires a database to work (one from the list of supported ones) then it should be present on the right side of the picture. Maybe they renamed it "state" in 3Factor?

[armincerf]

Right yes, state means the database. I think the left hand side is just a roughy description of what you might do normally (the 'wrong' way, according to this thing). I don't think it's supposed to be an old->new diagram or at least I didn't see it like that

[armincerf]

• https://nhost.io/ Project which shares similarities to Site (though using Postgres through hasura)
  Has a page discussing how they achieve custom logic (events like sending emails are done through hasura event triggers)

[reborg]

Nice set of questions. My current thinking below:

Where does custom logic run? (on the site server? on a service such as AWS Lambda?)

Running on the site server produces coupling between 2 separate concerns, scalability/availability of Site itself and scalability/availability of user functions. It also creates a problem of having to sandbox the evaluation (can I run something like (System/exit 0)?). So it seems better to evaluate elsewhere. AWS Lambda is surely a first candidate and probably should pave the way, but perhaps the desired evaluation backend should be pluggable.

How is the custom logic defined (both the API details such as endpoint/parameters and the code itself)

I see this as a Lambda entity which describes its reference ID, signature and return type. The lambda entity model in Site supports standard CRUD operations plus evaluation. Evaluation triggers in AWS Lambda by invoking the corresponding reference. We could think of some standard attribute that allows to connect the input from an XT query and push the output back into XT, if necessary. Otherwise the output could be returned synchronously as part of evaluating the /lambda endpoint (e.g. the client is appended waiting for it).

Where does the custom logic live (is it put into XTDB or does site have no opinions about this)

The custom logic itself, the actual body of the function, should probably live outside Site. Reasoning here is that the business logic would still need to be hosted on some file system to allow development and testing. Going back and forth to extract the current logic from Site would be a sub-par experience compared to Github.

How are event triggers (e.g send an email after a user is added) done, presumable need some queue

This could map into an initial POST /lambda/send-email to register the business logic reference with Site. The send-email lambda should be configured so the output of the evaluation (success/failure) is recorded into XT for later retrivial (in other word, an asynchronous call). A client would then require /lambda/send-email/eval either from the front-end or as post-transaction trigger from XT (I'm assuming there is something like this). The async eval returns an evaluation ID that can be checked at any time for success/failure.

How are the results from running custom logic used in an XT query (e.g fetch from some external source and query XT based on result)

I guess the lambda definition would need to allow input/output hooks into XT. Which means, instead of passing custom parameters, the input could be sourced from the evaluation of an XT query which is part of the definition of the lambda. Similarly for the output, the result could be stored directly into XT.

How are the results of running an XT query passed to some custom logic (e.g query something from XT and use it as a parameter in a custom function)

The lambda object in XT would need to be created with a query property, which is the query to issue to XT at the time of evaluation. The actual lambda function should expect a single parameter of type "collection" which represents the result of calling the query. Site would be responsible to call such query and pass it to the invocation of the lambda function.