tokio-util: allow encoding borrowed buffers at LengthDelimitedCodec

[MOZGIII]

Motivation

Fixes #6116

Solution

Implement <'a> Encoder<&'a [u8]> for LengthDelimitedCodec.
Call into <'a> Encoder<&'a [u8]> in Encoder<Bytes> of LengthDelimitedCodec.

[mox692]

As discussed in the original issue, this change could break the existing code like this, so I'm not sure we can make this change right now.

[Darksonn]

Closing as it is a breaking change.

[MOZGIII]

I think it would make sense to documenting that this is a permanent state and won't be fixed. The rationale for documenting this to express that avoiding breaking changes is prioritized higher than having an actual working parts there.
Also, with this setup I bet forks that fix this exist, and we might point the users at the discovery phase to that direction, such that they avoid stumbling upon this situation and wasting time like I did.

[Darksonn]

I certainly don't want to waste people's time. What kind of documentation would have helped you?

The issue you referenced has a comment from me saying that we will not fix it until we decide to release v0.8.0 of tokio-util.

[MOZGIII]

I suggest documenting it right in the code. The path to discovery comes from an issue of trying to avoid a copy when you have a &[u8] or Vec<u8> whereas the code requires Bytes. This is obviously a serious issue for some, but without a fix, and worst of all a clear message that this has already been looked and and considered won't be fixed.

Concretely, I suggest researching an alternative that doesn't suffer from this issue and adding a documentation section to the code that references that alternative.

---

Actually - I think there is a way to fix this - we can just add another type for the LengthDelimitedCodec (say LengthDelimitedCodec2 for example) that would have a fix and would not break people's code by being a separate type.

[MOZGIII]

Also, I usually release things and bump versions when there is stuff to release - and this breaking change pretty much is that, so to me this justifies the release of v0.8.0. Why wound't you cut the release? I just don't understand what is your decision making process is.

In my view, the releases should follow the need to release the functionality, not the other way around. If want to hold the release of 0.8.0 because you want some feature to ship at 0.7.x - well, this is not a good strategy because it can delay the release indefinitely and also because back-porting exists. Just for some context - the bug was reported at 2023 - but the issue itself existed for way longer than that.
I get that tokio release schedule exists and is quite slow - there was no breaking release of tokio-util even (although it is pre-1.0) for a quite long time - but there is no clarity on how the tokio-util specifically will evolve and when to expect breaking release.

On a side note - tokio-util is clearly a bunch of stuff that would perfectly fit into separate creates organized together for some reason - presumably ease of maintenance - but this is a particular case where a breaking release is due and it sort of seems like mixing functionalities like this in one create is perhaps not a good decision? I would be interested to learn more about how well-though was this. I - just like maybe everyone else in the rust ecosystem - have been following tokio news over the years, but never stumbled upon an reasonable justification for this.

[Darksonn]

I mean, it is probably about time for another breaking release of tokio-util. It's been two years since the last one. But it is not something I want to do often. Realistically, when I make a breaking change to tokio-util, that will result in many hundreds of other maintainers spending time on upgrading the version of tokio-util used in their projects.

As for why tokio-util is not split up into several smaller crates, the reason is indeed maintenance. The Tokio project used to be split into many smaller projects in the v0.1 days. This was a very large maintenance burden and confusing for end-users, so the decision was taken to merge them into a smaller set of crates, which resulted in the set of crates you see today. The split between tokio and tokio-util exists because breaking changes are impossible in tokio, but not in tokio-util.