Have materialized topics be compatible with the Kafka API

[graphcareful]

Currently as it stands, most aspects of the coprocessor implementation are not integrated within the Kafka API. For example, if you perform a request to list all topics, no materialized topics can be returned. Same case with offsets, consumer groups cannot work, partition movement, etc.

The source of all of this is due to the fact that materialized topics are not first class citizens. The controller has no information about them, for all intensive purposes they exist as flat files on each machine. Most if not all of the time to handle one of the above mentioned requests, one must query controller metadata to perform some type of lookup to handle the request. The way coprocessors currently work to get around this is to parse the materialized topic for the source topic and then query local information about the source topic. However this cannot work in all cases, and even if it did, this would mean adding edge case code in every request handler to handle a materialized topic request.

In this discussion I propose that materialized topics are treated as normal topics, with the only difference being a small flag within controller metadata to distinguish between the two. This would be necessary to prevent produce requests from producing onto these topics. If materialized topics are normal kafka topics, then by default all features that normal topics can do will now be available for use with materialized topics. Things like ACLs, consumer groups, partition movement, integration with the Kafka API, etc., will just automatically work, without any additional edge case code to handle them too. As an added plus integration within a higher abstraction will allow coprocessors to be able to consume from input topics only up until an offset that has been replicated across the cluster, making the outstanding truncation issue a non-issue.

[graphcareful]

The details of the solution are not discussed here and have not yet been decided. The non-trivial portion of this is actually deciding how to complete the above task in a way that still avoids all of the excessive raft chatter one would incur when transforming a topic without copro.

When created I will be compiling them together with our rfc format into a proper proposal

[dotnwat]

I fully support the goal of making materialized topics first-class managed objects in the controller. So many things will become safer and easier. This needs a full design document, but it will be totally worth it.