"Domain Data Liberation" - Enhancements needed?

[iolalog]

Hi everyone,

We are working on "Domain Data Liberation" these days. Most of our core domains (that handles business processes) are wrapped in REST API's, but we do not support event driven and analytical patterns well enough (nor file transfer for that matter). I will start with the questions, and then explain the background for those that are interested:

1. Query interface specification: Are there plans to crate a specifications similar to OpenAPI and AsyncAPI for typical "query interfaces"? By that I mean SQL (query databases and files), GraphQL (query graphs), SPARQL (query RDF graphs), MDX (OLAP cubes), LDAP (Active Directory) etc. I have found SwaggerQL that is "sort of" OpenAPI for SQL, but it is not maintained, and it is also lacking a lot of features that OpenAPI of today has: https://github.com/swaggerql/swaggerql
2. File transfer / storage interface specification: Like the previous question, just for file transfer and files. Thinking of SFTP, NFS, OneDrive and any other storage / transfer mechanism for files. If we had a specification that describes the interface and the schema for file stores, that would be neat. I have found no such specifications.
3. AsyncAPI links: You recently added this to OpenAPI. To give an example, the accountOwnerId in our (bank) accounts schema can now be linked to the customerId in our customer schema using this. Will this concept also be transferred to AsyncAPI? (so we can link events to each other the same way) Alternatively, should this be part of the JSON Schema instead of / in addition to?
4. Primary key, foreign key and business key(s) in OpenAPI / JSON Schema: While path parameter and similar can be said to the the primary key you need to "get a specific account", for customer this is typically a generated UUID. What we would like is the ability to specify the "primary key" attribute (say, customerId), the foreign key attributes (say, homeAddressId is a foreign key pointer to the address schema) and the business key(s) attributes (say, the uniqueness behind the generated customerId is actually a combination of customerType, publicId and customerCountry). This unmaintained JSON Table Schema has the primary key and foreign key concepts at least: http://dataprotocols.org/json-table-schema/#primary-key

Background
The idea is to do "Domain Data Liberation" as follows:

• REST API's are the main interfaces for our core domains. We document the interfaces with OpenAPI and the data with JSON Schema. These will be the basis for interface documentation and schema definition for our 3 other integration patterns (message / event interfaces, query interfaces and file interfaces).
• Whenever domain data change (POST, PUT, DELETE), an event containing the JSON Schema is sent to our EventHub (Kafka). We document the interfaces with AsyncAPI (generated from the OpenAPI) and the same JSON Schema (likely wrapped with CloudEvents to add some required attributes for describing source, id etc.).
• Events can then be landed in data stores fit for operational query patterns (GraphQL for instance) to support Command Query Responsibility Segregation (CQRS). The GraphQL graph would then be generated based on the OpenAPI and JSON Schema. Need something similar to OpenAPI to describe query interfaces (see question 1 at the top).
• Events can also be landed in data stores fit for analytical query patterns (Data Lake, Cloud Data Warehouse etc.). The table schema would then be created based on the JSON Schema. Need something similar to OpenAPI to describe query interfaces (see question 1 at the top).
• Events can also be landed in files stores for archiving and transfers. Need something similar to OpenAPI to describe file interfaces (see question 2 at the top).

The concept above should enable us to do polyglot data modelling (model once, reuse across multiple interfaces / data stores) and enable us to generate interface documentation for the 3 other integration styles based on the initial OpenAPI documentation.

So, to summarize, we would have the following:

• Domain data interfaces are described with similar open specifications (OpenAPI, AsyncAPI, , .
• Domain data is documented and modeled once (JSON Schema) and reused across all 4 integration styles and various data stores. We do a lot of double work with domain data to support operational and analytical workloads today.
• Interfaces documentation can be imported into one or more catalogs to enable humans to discover them (search, browse etc.) and machines to find each other (service discovery).

I will split this up later based on initial feedback, and probably take some of it other places, but thought I would start in the OpenAPI community since OpenAPI will be our "master" in the setup suggested above.

Regards,
Olav

[handrews]

@iolalog no idea if you're still interested in this- the project was somewhat in hibernation during much of the pandemic. To answer your questions:

1. Query interface specification: Are there plans to crate a specifications similar to OpenAPI and AsyncAPI for typical "query interfaces"

No, although the "signatures" concept in OAS 4 "Moonwalk" might be relevant, so you might want to join the discussions in that repo. I doubt we'd consider a specific query language in-scope, but Moonwalk's approach to endpoints, query parameters, headers, and request bodies might make it easier for you to build such a thing on top of it.

2. File transfer / storage interface specification: Like the previous question, just for file transfer and files.

The OpenAPI Initiative is focused on HTTP APIs, so I'd say these things are out of scope for us.

3. AsyncAPI links

AsyncAPI is a completely separate project, so you'd have to ask them.

4. Primary key, foreign key and business key(s) in OpenAPI / JSON Schema

OpenAPI 3.1 added full JSON Schema compatibility so we could get out of the business of extending JSON Schema ourselves. In Moonwalk there may be a more modular approach allowing for JSON Schema alternatives. So I'd either look at that in Moonwalk, or take your question to the JSON Schema folks.

Of the things that are in-scope, I think they would have to be addressed in Moonwalk, so I'm going to close this discussion (this repo is now focusing on the 3.x releases) and encourage you to participate in the OAI/sig-moonwalk discussions if you'd like to pursue this further.