Implement BufRead/Write for en/decoders alongside Read/Write

[chernoivanenkoofficial]

Hello.

Have been using deflate for encoding/decoding various moderatly sized data (under 5KB), where I need to parse some encoded heading variabely-sized metadata first.

Have been wondering, if there is any strong reason as to why encoders and decoders do not implement BufRead/Write, if internal implementations certainly use buffered IO.

Even though files in question are small, they can quickly bloat in numbers, and the neccessity of repeatedly either allocating in-memory raw buffers on heap or wrapping decoder/incoder in BufReaders/Writers (which, while enabling more "lazy", "stream-fashioned" approach, still do those same allocations internally) seems concerning perfromance-wise.

[Byron]

Thanks for sharing!

I hope you don't mind me turning this into a Q&A due to its format. However, I also pinned it for greater visibility.

That way those in the know might be able to shed some light on this - usually there are very good reasons for the API as is even though these are sometimes not easily discovered.

[jongiddy]

It is is not a good idea to have a single type implement both Read and BufRead. Either:

• the Read implementation uses the BufRead implementation and Read operations copy data twice: once to fill the BufRead internal buffer and once to fill the caller-provided Read buffer; or
• the BufRead implementation uses the Read implementation and the caller can use a BufRead operation to fill the internal buffer and consume part of it, and then use a Read operation that skips data in the BufRead buffer.

We could provide two types that implement each trait (similar to how flate2 handles the input stream). But there are already 3 encoder types and 3 decoder types for deflate, so I'd like to see more evidence of a benefit over using BufReader<Encoder>.

Note, neither trait requires repeated allocations. BufReader allocates a single buffer and reuses it. Read can use the same approach, but managed by the caller.

[chernoivanenkoofficial]

I'll rephrase my question a little.

If deflate uses internall bufferring when decoding/encoding, when is it not beneficial (or at least possible) to expose it a little more through buffered IO operations?

Of course, it has sense only if internal buffer is of relevant size (in KBs values). And this is of course a little tricky, as the problems you listed are on point and it's not a good idea to use either BufRead or Read implementations for implementing the other one. And it even possible requires some overriding for BufReader member's default implementations. But as inflate, for example, does buffering internally, but doesn't provide BufRead-like utility (like read_line() or read_until()), it seems upsetting.

[chernoivanenkoofficial]

Having spent more time on looking into the code, I've found out, that only unprocessed bytes from underlying stream are buffed. Output proccessed bytes are decoded on demand in quantities requested (size of output buffer) at most. Thus, wrapping inflate structs in BufRead doesn't do any processed bytes unneccessary copies, if used with minimal common sense (that being e.g. not using buffering, if the end use-case is calling read_to_end).