High memory usage when compressing data blocks with up to 2MiB sizes with zstd.Encoder.EncodeAll()

[valyala]

It has been appeared that every zstd.Encoder instance occupies around 500MB of RAM when it is used for compressing data blocks with up to 2MiB sizes each by concurrent goroutines calling zstd.Encoder.EncodeAll() - see this comment for details.

IMHO, 500MB of RAM per every zstd.Encoder instance in this case sounds too much, and its memory usage can be reduced somehow.

[klauspost]

Higher levels take more memory - not only by bigger default window size.

Tips:

A) Use less instances. You seem to do that somewhat, but AFAICT you are indexing by original level, not the level used internally. Use the returned value from idx := zstd.EncoderLevelFromZstd(compressionLevel) to get your index.
B) Limit Window Size.
C) Use WithLowerEncoderMem
D) Limit concurrency - maybe on 32 bit platforms.