API patterns to serve editorial UI and others

[jardakotesovec]

Introduction

While working on new submission listing, there is lots data both in submission listing itself and in the submission summary.

Opening this discussion to explore what API design principles are good fit for our use cases.

We are facing typical challenges with REST api when serving the UI, these are often approached with one of the following strategy:

1. Making lots of calls to different API endpoints from UI and combining data in the browser
2. Making dedicated API endpoints for given screens, which provides all (or most) data in one response

In the 'submission listing UI' there are two major screens. Table with list of submissions and submission summary with submission details. Lets dive more into how we deal with these challenges on these.

Submission listing

Currently we leverage private endpoint _submissions, which pretty much falls under strategy number (1). Everything that needs to be displayed in the table and in the review assignments popup are included in this one endpoint.

This was intentionally decided to have good performance and UX. Otherwise we would have to make separate API call when Review Assignment popup is clicked on, which would delay each of these popups and make the overall listing not responsive enough.

Submission summary

When submission summary modal is opened only submissionId is passed (check Note1 for more details). And submission modal starts fetching metadata from various endpoints. And ideally if I would want to avoid over fetching, it would be following waterfall

1. get submission from submissionId & get participant based on submissionId & fetch files based on submissionId (in parallel)
2. get currentPublication from publications/id endpoint, once I know the id from fetching the submission
3. fetch issue details once I know the issueId from currentPublication & fetch section detail once I know the sectionId from currentPublication (parallel)

This is one way to do it with existing API. Its fine to fetch various metadata from multiple endpoints (and there are some benefits to it as we display different metadata based on the stage and role and omp/ojs/ops). Concern in this case that it would require 3 round trips in sequence, which adds up in the latency.

Here are some patterns I can think of, maybe you can think of some additional ones:

1. start using query param 'includes' which give additional control which dependencies to include in the response. In my case the request would like like /api/submission/12?includes=currentPublication.issue,currentPublication.section,currentPublication
2. having additional endpoints like /api/submission/12/currentPublication/issue, /api/submission/12/currentPublication, so I could fetch from these in parallel just from knowing the submissionId
3. Similarly to 1) we would include some of the dependencies, but we would not make it configurable via query param and just careful decided which one to include so it meets the consumers requirements "most of the time"
4. Creating dedicated /api/_submission/12 endpoint for serving the summary page which would include whats needed for that page.

My opinion

I think option 1) could work quite well. Query param would not support all dependencies obviously, but some particular which we find important.

I think that could help provide path to also consolidate the public (/submissions) and private (/_submissions) end points to one more flexible public one, where we could control how much dependencies to include to avoid over-fetching for example review assignments if the consumer does not need them. End result could be used by submission listing as well as other consumers (integrations, plugins).

And also option 2) is good for some cases - its already used for things like participants or files . Especially for more complex dependency, where additional query params are good for filtering the results.

There also will be considerations from the implementation point of view, where some fields and API are relevant only for some apps (ojs vs omp vs ops), so that also might affect which of these patterns is better fit. I don't have much of insights on that.

Notes

Note1: If you are wondering, why not using available submission metadata from submission listing in submission summary to reuse already fetched metadata. This assumes that the submission in summary is being currently listed in the table. And there are some edge cases when this might not be true: table will be refreshing data regularly to display current state therefore submission might slip off the listing. And more importantly there will be lots of jumping between decision page and submission summary, and when redirected back its easy to keep the submissionId as query parameter and start loading submission summary details based on that, rather than hope it will first be fetched in the table . So technically it could be considered as performance optimisation, but at this point I rather keep it simple and independent from each other.

[jonasraoni]

The codebase has something like the #1 in place (regarding including extra information), but it doesn't smell very good to me. The #1 also reminds me about the JSON API pattern.

I think the #1 will fit well for now. The fields returned by the API are controlled by a JSON file (see:

pkp-lib/schemas/context.json

Line 30 in 363a145

"apiSummary": true,

), so it could be overridden to return more data.

---

For the long term, the GraphQL discussion deserves consideration, it would address such issues in a more standard way.
I think we don't have enough resources to maintain both a REST and a GraphQL API (as well as coming up with new and different patterns on every release), so it's good to discuss here and settle for something solid (agreed by the majority of members).

[asmecher]

Agreed with #1 too, with nesting driven by need (e.g. a bottleneck that needs resolving) rather than speculatively attempting to get head of requests.

Jarda and I spoke a little about other options:

• Client-side caching. This will have the usual cache invalidation problems, and will not be broadly useful because we still don't have a comprehensive single-page app approach. And browser storage is not good enough.
• Querying more data to permit parallel fetching, e.g. getting all sections or all issues rather than waiting until we know which one we need. I think this could represent a lot of wasted data, so not a great approach.