[Bug]: Split package preview and split compress package bug

[psvajaz]

bit7z version

4.0.x RC

Compilation options

BIT7Z_AUTO_FORMAT, BIT7Z_USE_NATIVE_STRING

7-zip version

v22.01

7-zip shared library used

7z.dll / 7z.so

Compilers

MSVC

Compiler versions

MSVC2017

Architecture

x86_64, x86

Operating system

Windows

Operating system versions

Windows 11

Bug description

1、Unable to obtain the file list of the partitioned compressed package using the BitArchiveReader class
2、When split compressing by package , if the number of packages exceeds 999, an error will be reported

Steps to reproduce

No response

Expected behavior

No response

Relevant compilation output

No response

Code of Conduct

• By submitting this issue, I agree to follow bit7z's Code of Conduct

[rikyoz]

Hi!

1、Unable to obtain the file list of the partitioned compressed package using the BitArchiveReader class

Did you open the archive using BitFormat::Split or with the specific archive format?
In the first case, BitArchiveReader will only report one item: the whole archive; this is the expected behavior.
On the other hand, if you used the specific archive format, then it's a bug! Are you trying to open more than 999 volumes also in this case?

2、When split compressing by package , if the number of packages exceeds 999, an error will be reported

I can reproduce the error. I'll fix this bug as soon as possible!

[psvajaz]

I used BitFormat:: Split for preview and obtained content with only one item. In this situation, how should I continue previewing the content of this item?

I think that in this situation, Bit7z should automatically help me obtain the content of this compressed package. Do you think this is more reasonable

[rikyoz]

I used BitFormat:: Split for preview and obtained content with only one item. In this situation, how should I continue previewing the content of this item?

Unfortunately, it's not possible when opening the archive using BitFormat::Split.

I think that in this situation, Bit7z should automatically help me obtain the content of this compressed package. Do you think this is more reasonable

Bit7z does support automatically reading the content of the compressed package in a multi-volume archive, but you have to specify the compression format, rather than BitFormat::Split.
For example:

BitArchiveReader reader{ lib, "archive.7z.001", BitFormat::SevenZip };
// reader will directly read inside the whole 7z archive

[psvajaz]

Thank you

[psvajaz]

https://www.7-zip.org/a/7z2201-extra.7z
I used this compressed package, decompressed it, and recompressed it using the official 7z tool. The compressed file type is 7z, and the partition size is 1kb. After compression, I obtained 1049 files for each partition. When using BitInputArchive to load the compressed package and debugging the bit7z source code, it was found that the while loop in CMultiVolumeInStream:: CMultiVolumeInStream loops to volume_index=510, the CFileInStream:: open method will fail to open the file, and the test environment is x86

[psvajaz]

https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/setmaxstdio?view=msvc-170

[psvajaz]

Execution After _setmaxstdio (8192), when previewing compressed packets with more than 999 volumes, normal operation is possible
There are two issues with the test now:

1. When compressing in separate volumes, a maximum of 999 split packages can be generated
2. When decompressing a compressed package with more than 999 volumes, it cannot be decompressed properly

During these two days of debugging, it was found that the Stream could not be released normally, causing IO abnormalities

[rikyoz]

I used this compressed package, decompressed it, and recompressed it using the official 7z tool. The compressed file type is 7z, and the partition size is 1kb. After compression, I obtained 1049 files for each partition. When using BitInputArchive to load the compressed package and debugging the bit7z source code, it was found that the while loop in CMultiVolumeInStream:: CMultiVolumeInStream loops to volume_index=510, the CFileInStream:: open method will fail to open the file, and the test environment is x86

Execution After _setmaxstdio (8192), when previewing compressed packets with more than 999 volumes, normal operation is possible

Thanks for the debugging!

Yeah, I actually noticed this issue while fixing the bug you initially reported here.

The main problem is that bit7z keeps all the volume streams open until the compression/extraction operation finishes. This is actually the same behavior that 7-zip had before the latest v23.01 (still not supported by bit7z, by the way, but I'll work on it).
On Windows, the default limit is 512 opened file handles, hence the error when trying to operate on archives with many volumes.

Yesterday, I pushed two commits to the develop branch (4121bff and 8a0c1c9) that fix the problem, at least when compressing.
This is just a temporary workaround that uses _setmaxstdio, as you suggested.
I'll also push the same fix for CMultiVolumeInStream as soon as possible.

Ideally, the library should keep fewer streams opened, but this would require more code changes, so a cleaner fix will probably come after the v4 stable release.

When compressing in separate volumes, a maximum of 999 split packages can be generated

Isn't this the same issue you initially reported? If so, I already fixed it on develop (260ebe0).

When decompressing a compressed package with more than 999 volumes, it cannot be decompressed properly

I'll check this, thanks!

During these two days of debugging, it was found that the Stream could not be released normally, causing IO abnormalities

Could you give more info about this issue?

[rikyoz]

Quick update: I just pushed a commit to develop that should fix the opened files limit issue also for CMultiVolumeInStream (e787c3d).

[psvajaz]

OK, Thanks. I will Test

[psvajaz]

compress and decompress 999 split packages the problem still exists.

[psvajaz]

preview is ok

[rikyoz]

Unfortunately, this time I cannot reproduce the problem. In my tests, bit7z handles archives with more than 2000 volumes just fine.

[psvajaz]

Can you tell me if you can call the code for decompression and compression methods and let me see if it's my coding problem

[rikyoz]

Compression:

BitFileCompressor compressor{ lib, BitFormat::SevenZip };
/* Setting the volume size as 1/2048 of the uncompressed file size;
   obviously, we will have less than 2048 volumes due to compression, 
   but they will still be more than 2000, at least in my case! */
compressor.setVolumeSize( file_size / 2048 );
compressor.compressFile( "path/to/file.exe", "multi_volume.7z" );

Extraction:

BitArchiveReader reader{ lib, "multi_volume.7z.001", BitFormat::SevenZip };
// Output folder which will contain the original unsplit and uncompressed file.
reader.extractTo( "output/" ); 

alternatively

BitFileExtractor extractor{ lib, BitFormat::SevenZip };
extractor.extract( "multi_volume.7z.001", "output/" );

[psvajaz]

This error will be reported when the number of questionnaires exceeds 8188