issue with query in deta base with 20000 rows

[ristomitts]

Hi. I have created a deta base with about 20000 rows. They are all inserted with a unix timestamp and for debugging i added year, month, day integer fields also. The years range from 1990->2023. And I added them to the database from oldest to newest based on date.

I notices when I query something with like {year:2020} or {year?gte:2015}, I do not get any results. While query {year:2000} return results as expected.

When debugging this I noticed that the result i get from {year:2020} for example is:
{items:[], count:0, last="somekey"]

So I made a loop while last key was present and then finally got the result i wanted after 6 loops.

Is this expected behaviour when using querys or is this a bug?
I can get it working with a last key loop. But it seems a bit unintuitive to loop through a list of empty responses until the data appears in the last request.

For example when using the UI to query you cannot get any results at all when querying items that are stored last in the database.

[aavshr]

hello @ristomitts this is expected behavior (although inconvenient sorry).

Base scans through upto 1 MB(without applying any queries) and then applies the query your provided. If the 1 MB scanned data does not have any items that match your query, it returns this reponse with no items and a last key scanned (meaning further items are to be scanned).

We realize this is not ideal. This is due to a technical limitation at the moment and we plan to mitigate this issue as soon as we can.

[ristomitts]

As long as I know how it behaves I can work around it. Thank you for the answer.

[kikogassen]

This behavior would be so great! I'm developing an Android application using Deta and this limitation is the only point that isn't covered by your great system ;)