TTL=0 data aka flow-only data ingestion

[sunng87]

What type of enhancement is this?

API improvement

What does the enhancement do?

From one of our key user who only wants to generate derived data without persist original data. We will need to provide a special case for this type of TTL=0 data: feed it to flow and drop it before entering memtable.

Implementation challenges

No response

[killme2008]

I think we can design a virtual table for such cases just like

create table monitor(ts timestamp time index, cpu double) with(engine='virtual');

It serves solely as an ingestion endpoint and a source for the flow engine, but cannot be queried.

[sunng87]

I think ttl=0 is a better abstraction. We'd better not to introduce new concept unless it's necessary.

[killme2008]

I agree, but if we can implement the ttl=0 without consuming any memory, that would be better. Just drop the write requests before they reach the memtable, maybe we don't create any memtables for such tables.

[killme2008]

This feature should be prioritized as another user requires it.