Fix RPDE implementation

[nickevansuk]

Applying rules:

• Include afterId and afterTimestamp parameters in the next URL (and check the underlying query) ( Include either both afterId and afterTimestamp parameters in the next URL, or afterChangeNumber developer-microsite#32 )
• afterTimestamp and modified must be integer ( afterTimestamp and modified must be integer  developer-microsite#31 )

Please do check the underlying query is being implemented correctly: https://www.openactive.io/realtime-paged-data-exchange/#sql-query-example-for-timestamp-id:

  --include WHERE clause only if @afterTimestamp and @afterId provided
      WHERE (modified = @afterTimestamp
               AND id > @afterId)
         OR (modified > @afterTimestamp)
   ORDER BY modified,
            id

Not like this:

{
  "next": "http://api.letsride.co.uk/public/v1/rides?from=1970-01-01+01%3A02%3A10",
  "items": [
    {
      "state": "updated",
      "id": 2710,
      "kind": "event",
      "modified": "1970-01-01 01:00:01"
      "data": {}
    }
  ],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}

Like this:

{
  "next": "http://api.letsride.co.uk/public/v1/rides?afterTimestamp=1453931925&afterId=2710",
  "items": [
    {
      "state": "updated",
      "id": 2710,
      "kind": "event",
      "modified": 1453931925,
      "data": {}
    }
  ],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}

Note that the next URL must include the id and the modified values from the last item in the current page (see here for more details: https://www.openactive.io/realtime-paged-data-exchange/#modified-timestamp-and-id)

[sevenpointsix]

This should be ready for testing at http://lr-api.staging.phoenixdigital.agency/public/v1/rides/

[nickevansuk]

Just looking at these pages, in order:

• http://lr-api.staging.phoenixdigital.agency/public/v1/rides?afterTimestamp=1511529586&afterId=1805
• http://lr-api.staging.phoenixdigital.agency/public/v1/rides?afterTimestamp=1477924270&afterId=87872
• http://lr-api.staging.phoenixdigital.agency/public/v1/rides?afterTimestamp=1480753800&afterId=88121

Note that the ORDER BY modified, id means that afterTimestamp should be increasing, which they are not.

Also the second page of the feed returns "No ride data found".

And this page returns an error.

These indicate that the query has not been implemented correctly. Please check:

  --include WHERE clause only if @afterTimestamp and @afterId provided
      WHERE (modified = @afterTimestamp
               AND id > @afterId)
         OR (modified > @afterTimestamp)
   ORDER BY modified,
            id

[nickevansuk]

This is much better! However on this page the next URL doesn't appear to have an afterTimestamp which is greater than the current value, which suggests the query might still be wrong.

There's now a helpful video on this page which might be useful (any feedback welcome!)

[sevenpointsix]

Sorry @nickevansuk , I've tweaked this now, let me know if it's resolved?

[nickevansuk]

That's great, and the validator now works for this since your fix too!

http://validator.openactive.io/rpde?url=http://lr-api.staging.phoenixdigital.agency/public/v1/rides/

Note the issue in the validator relating to the last page - still one to fix there

[sevenpointsix]

Hi @nickevansuk . This is the final issue to fix I think! Our final page returns the message 'No ride data found' as a string, which is incorrect. How should we be rendering an empty page of data? Or, should we be excluding the "next" link on a page that has no valid "next" page?

[nickevansuk]

If you check our the video in the earlier comment the last page stuff is explained near the end :)

Basically just has zero length array for items and the next page URL equal to the current page

[sevenpointsix]

Should be working now I think @nickevansuk ?

[nickevansuk]

Just had a quick look... still errors... try running the validator and looking at the bottom of the list of errors?

http://validator.openactive.io/rpde?url=http://lr-api.staging.phoenixdigital.agency/public/v1/rides/

It looks like the last page still doesn't conform (e.g. http://api.letsride.co.uk/public/v1/rides?afterTimestamp=1536246355&afterId=27)

Also if you make the limit of items on items returned in each page 500 you'll get less warnings in the validator too.

[sevenpointsix]

I can't replicate this @nickevansuk , I only see warnings and no errors. Is it definitely throwing an error for you if you re-run a full validation?

[nickevansuk]

Sorry I think you've found another bug in the validator, will investigate that...

Try this: http://validator.openactive.io/rpde?url=http%3A%2F%2Flr-api.staging.phoenixdigital.agency%2Fpublic%2Fv1%2Frides%3FafterTimestamp%3D2001%26afterId%3D8402

Also this page returns an error, but should look like a last page:
http://api.letsride.co.uk/public/v1/rides?afterTimestamp=1536246355&afterId=27

[sevenpointsix]

That second link is the live site @nickevansuk , we've fixed it on UAT (http://lr-api.staging.phoenixdigital.agency/public/v1/rides?afterTimestamp=1536246355&afterId=27). I've also fixed the issue found by the validator above.

[nickevansuk]

Excellent! Were you able to adjust the page size to 500 too?

[sevenpointsix]

The feed should now be returning 500 items per page @nickevansuk

[nickevansuk]

http://lr-api.staging.phoenixdigital.agency/public/v1/rides?afterTimestamp=1526246355&afterId=27

Found this just now too...

[nickevansuk]

Seems to be resolved