ROM packages support

[kcgen]

A ROM package holds the game's files and metadata inside it.

In order to elegantly support this, DOSBox's existing file IO calls should be wrapped on a 1-to-1 basis, sending them through an IO backend.

The IO backend can then operate on files on the host (as the calls current do), in the writable overlay area, or held in the ROM package.

DOSBox has thousands of IO calls, So this issue tracks the effort of simply wrapping such calls, so when we finalize the ROM package's container and compression format we will be ready to simply drop it into this IO backend.

IO calls also exist inside 3rd party code (such as the audio codecs, MT32, and FluidSynth) that we currently use when we pass those libraries a filename and let them perform IO calls themselves. To avoid those, we will need to use load-from-memory type API calls (allowing us to perform the file read beforehand), or pass them IO callbacks, which some libraries provide, such as the CD-DA codecs.

This issue allows us to discuss details of how we want to go about this.

[shermp]

So, I've been thinking about this today, and was thinking one idea might be to take inspiration from how EPUB files are packaged as a conceptual model.

As a background, I'm heavily involved in ebooks, so I'm rather familiar with the basics of EPUB packaging.

The EPUB format is a standard zip archive. There is one standard, uncompressed file stored in META-INF/container.xml. This contents of this file specifies the location of the default OPF file, and any additional OPF files (although I've never seen an EPYB in the while that actually has multiple OPF files).

The OPF file is a combined ebook metadata, file manifest and (because it matters for ebooks) xhtml file order. Any files not listed in the manifest are ignored. Apart from the mandatory META-INF/container.xml file, there is no mandatory file/directory structure.

Obviously, any potential Dosbox format would probably be quite different technically. For example, much more aggressive compression. I notice that ZStandard is now an official compression algorithm allowed in ZIP archives for example. And Dosbox would probably use it's own metadata and manifest formats.

[shermp]

Help! I've been thinking again, and had an idea. Note the following is very much an initial idea, and would require a lot of refinement.

Idea: A SQLite based container format

This is an idea to create a 'ROM' format using SQLite as the container.

Why SQLite? It is a very stable, well tested, single file SQL database which allows plenty of flexibility as a file format. The SQLite library is available for basically every platform under the sun. One file format that uses SQLite as its container is MBTiles, a file that can contain millions of map tiles.

SQLite would allow anybody to easily "peek under the hood" using standard SQLite tooling.

The following is the beginnings of a proposed format schema.

Metadata Table

CREATE TABLE metadata (
    format_version INT,
    json_metadata VARCHAR
)

Store metadata about the game or application. Idea is to keep it as a json file for easy extendability. Maybe include minimum versions of dosbox supported etc.

Config Table

CREATE TABLE config (
    id INT,
    name VARCHAR,
    config VARCHAR
)

Store one or more dosbox configs. The first config would have ID 0, and be the default config. Additional configs would allow for variations such as different sound and graphics options. Dosbox would load one of these configs when the file is opened, potentially allowing the user to specify which config to load.

File Table

CREATE TABLE file (
    id INT,
    path VARCHAR,
    mimetype VARCHAR,
    uncompressed_size INT,
    uncompressed_hash VARCHAR,
    uncompressed_chunk_size INT,
    num_chunks INT,
    compression_type INT,
    uncompressed_hash VARCHAR
)

Store a listing of all files in the container. This table determines how a file should be mounted, and how it's contents are stored in the file_content table. Files are stored in fixed size chunks with optional compression. The use of chunks will allow seeking to specific offsets in the file without having to decompress the entire file into memory. Multiple compression types are supported within a single container, to support the most efficient compression of different file types.

File Content Table

CREATE TABLE file_content (
    file_id INT,
    variant INT, --config id, support multiple variants of the same file
    chunk_num INT,
    chunk BLOB
)

Store the contents of files. A file may have multiple variants which correspond to the loaded config. Each chunk when uncompressed is the size specified in the file table.

[johnnovak]

My view: I'd only use SQLite if I need searchability, e.g. in a front-end that stores information about a list of games. For a single artifact, the use of SQLite is not a good fit and seems like overkill.

The simplest thing to do for a single game is to store it as an uncompressed ZIP file, then put extrra metadata files into it with know fixed pathnames.

Supporting multiple configs can be done similarly; dedicated config path subdir inside the ZIP, then the master metadata JSON could contain an array of config objects with pathnames and descriptions. And so on...

I don't see how the complication added by storing all this in SQLite is worth it when a simple JSON file approach would do the job perfectly well. "Just do the simplest thing that does the job (TM)"

About compression, I don't see much point of it at the package level. I'd mandate the use of uncompressed ZIP files myself, so then we can simply read the contents directly. Of course, if we had CHD support, the CHD image itself would be stored in the ZIP file uncompressed (as in not ZIP compressed), and the CHD reader could read the CHD directly, uncompress it on the fly, etc.

My points:

• Storage is cheap, and DOS games are generally not big, so who cares (unless you're hoarding games, in which case... buy more storage 😄)
• We could always double-compress for distribution;
• Yes, there are multi-CD DOS games, and what are they without exception? FMV games! Those CDs are filled with MPEG streams that you can't compress further easily.
• Single CD games that have PCM audio tracks similarly don't compress well. With FLAC you can achive 50% compression, but again, see point one 😄
• Compression could be beneficial when distributing games through the network, that's true. But it still wouldn't help much where it would be needed most, which are those multi gigabyte, multi CD games. I'd say it does not matter that much with the majority of games that came on floppies and therefore are in the sub 50 Mb range (usually well below that). But we can always double-compress; our uncompressed ZIP game package would have a dedicated extension (.dosgame or whatever), then we could compress that one more time with actual ZIP compression just for distribution.

[mkruer]

I propose that we revisit the fundamentals and draft a set of requirements for our request. It is important that we draft a comprehensive set of requirements to guide our efforts. While I have some initial ideas on how to achieve our objectives, it is essential to approach them with a degree of caution. To provide clarity, I will break down this proposal into four distinct components:

1. .dosrom: This file functions as the ROM image for a game, serving as the core element of the request.
2. .dosconf: Within this file lies the configuration data essential for DosBox, specifically tailored for the ROM file's operation.
3. .dossave: Acting as a storage repository, this file is designated for the preservation of in-game save data.
4. .dosplay: Presented as a concept to support the TAS, recording and replay of games without user input.

.dosrom:

It serves as the ROM image for a game. Once created, it should remain untouched unless there's a valid reason to modify it. When it comes to packaging, we must consider that many DOS games were modified during setup or initial runs. Within other console emulators, each game version became its own ROM, leading to cases where there might be 20 ROM releases during its run. To put it another way, there are 880 Sega Genesis ROMs, but there have been 5872 ROM releases.

While it's not the responsibility of DosBox Staging, should we acknowledge that some users desire a complete copy of a game from its original release to the latest user-made patch, all packaged together? Should we consider internal reversioning for the ROM? For instance, Sierra On-Line's game "Hero's Quest" had significant patches that altered gameplay. Additionally, there were compatibility issues between 5.25-inch disk saves and 3.5-inch disk saves due to resource file splitting during packaging. Furthermore, the game underwent at least five patches and a rebranding during its initial run.

There's also the concept of post-vendor patching, where unofficial fan patches are applied. Games like "Quest for Glory 3" were released incomplete, where in-game classes were unable to reach 100% completion. The fan community has taken it upon themselves to rectify these issues, but the patch requires the original game files. I see this as a trend that will continue.

I suggest that whatever format the ".dosrom" file should take, it should support basic reversioning options mentioned earlier, via support for Hard Links or symbolic links to remove duplication inside the ".dosrom" file. While I agree that space is cheap, I don’t foresee someone wanting to save or transfer an additional 5GB for video or audio data that is the same between patches of a DVD.

If agreed, we may want to come up with a way to constantly craft the ".dosrom" file to maintain consistence, a project space where we keep a SHA value and directory structure of all the files and then submit them to the project?

.dosconf:

This file serves as configuration data for the ROM file. While not mandatory, it can be optionally distributed alongside the .dosrom. It contains additional metadata for DosBox or other settings pertinent to customizing the game within DosBox. This includes but is not limited to:

1. Visual Settings
2. Sound Settings and ROM files
3. Performance Tweaks

.dossave:

The .dossave serves as a repository for storing save game data or any other configuration data generated by the program within DosBox.

To address patch-related save issues, such as those encountered with Hero's Quest, which suffered compatibility problems between 5.25-inch disk saves and 3.5-inch disk saves due to resource file splitting during packaging, the .dossave file should support multiple save hierarchies. This can be achieved by associating each save with the corresponding patch version of the game.

Moreover, it should provide the capability to overcome save limitations within a game by incorporating a method to manage split save data. For instance, in games like Wing Commander, which offer only 8 save slots, players often duplicated save files and swapped them to access older saves. To accommodate this, there should be an option to load additional saves (replacing the SAVEGAME.WLD), which could be seamlessly integrated into the OSD as an extra feature.

.dosplay:

I'm presenting this concept because I'm a supporter of the TAS (Tool-Assisted Speedrunning) and speedrunning community. If DosBox were to attain 100% reproducible RNG or the capability to inject pseudo-RNG for games utilizing RNG, it would become possible to record a gameplay session and precisely replay it, maintaining a one-to-one correspondence with the original.

[johnnovak]

What a thoughtful writeup of the various more subtle issues that makes the job of creating a universal DOS game pack format really hard! I'm also impressed by your in-depth knowledge of the Quest for Glory series @mkruer.

Given the difficulty of the task and that we're a small team of a handful of volunteers, and there is already a lot on our plate for the next 5 years or more, I don't see us becoming the inventor and maintainer of neither the pack format nor the database of games.

I can see us adding support for such a game pack format once it is established and proven to work well in practice, covering all or most cases you outlined. Frankly, the only realistic way for this to materialise is if a new team of people started a sister project and worked together with us on it. It's just too much for the current core team I think, but correct me if I'm wrong guys.

Then there is the TDC and eXoDOS collections that people can use right now. Decades of work went into those projects, so if someone is in a good position to work on a such universal pack format that caters for all possibilities, it's them. But my understanding is they're not too keen on repackaging their games into a ROM format (for various reasons), so probably this task would need a new team.

I think of this as IPF disk images. Countless hours went into designing the format and creating the hardware and software able to create those IPF images. Then decades of work to collect, categorise, and test the IPF dumps.

The job of an emulator author adopting the IPF format is much easier, almost trivial in comparison. I see us, the team working on the core emulator itself, as the adopters of such DOS game pack format invented and maintained by others as it's a huge undertaking.

[mkruer]

Thank you for your feedback. My intention was to highlight the challenges associated with archiving legacy software comprehensively. Often, the crucial step of defining thorough requirements is overlooked in favor of more engaging tasks.

Reading your comments, I have overlooked a potential and crucial aspect regarding full ROM based archive. This was brought to light when looking at the IPF disk image format. The proposed format should ideally also support preservation back to the original installation medium.

Regarding the TDC and eXoDOS collections, although valuable, they may have been modified to ensure compatibility with emulators, which raises concerns about fidelity.

If a project were to commence, the initial focus should be on clearly defining all requirements while remaining open to adapting to unforeseen obstacles. Subsequently, efforts would shift towards developing the necessary methods, tools to create the ROM images.
Interestingly, in reading a bug for the Roland MT-32 ROM, it sparked the idea of implementing a repository for hashing game files and structure and to establish a centralized repository to facilitate the creation of a verifiable ROM image. With the repository created, porting to a new format becomes much more trivial.

In regards to my extensive familiarity with Hero's Quest/Quest for Glory, I must admit that as a completionist, dating back to the original 10-disk 5.25 version 1.001, I've played the game countless times to achieve 100% completion. Unfortunately, due to save game failures resulting from alterations, I am eagerly seeking a version of the game that remains stable and doesn't falter every time I wish to revisit past endeavors.

A compelling argument could be made that the next feature to be added should be the ".dosplay" function. As long as the game supports reproducible RNG or has the capability to inject pseudo-RNG, the ".dosplay" feature can be utilized for generating test cases. This can include screen shot comparisons or identifying floating-point errors that may manifest with varying results.

The feature I am most looking towards in the immediate future would be the OSD as this has the greatest potential stuff like

Finally, if all the is taken away by this is keeping in the back of the project devs minds that at some point in the future they would like to implement some or all these endeavors, then I will consider this a success.

[idanmiara]

Have you considered adopting this?

https://github.com/schellingb/dosbox-pure?tab=readme-ov-file#load-games-from-zip