Game uses crazy binary data files about raceintospace

[rnyoakum]

I've gone through RIS and eliminated the reliance on the .but and .img files that were hanging around (merged via commit 49a85a5). Those have been replaced with pngs that had been extracted previously by the RIS team or that I extracted. I haven't deleted the original files yet.

[willglynn]

We've made a bunch of progress on this. Random data files have been replaced with PNGs one piece at a time, and #100 gives us a way to manipulate the graphics system independent of these crazy formats.

[hweimer]

With serialization via cereal being included, it is possible to make progress on this. In many cases, the game uses files containing a big binary array of some internal data structure. For example, the file mission.dat is one big array of struct mStr. So if you want to get the mission with code 42, the game will seek() to 42*sizeof(struct mStr) and read the next bytes into the proper struct. There are several possibilities how we can proceed to convert these files into human-readable JSON files:

1. Load all data files into memory when the game starts. Will significantly increase memory usage, but it's not 1992 anymore.
2. Perform live deserialization of the required data file. Might be too slow for some of the bigger files, although rapidjson is pretty fast.
3. Split the file into many small pieces, each containing one element of the array (e.g., mission_42.json). More cumbersome to work with these files.
4. Keep the existing data formats, but automatically generate the datafiles from the corresponding JSON files during compile-time.

Opinions?

[peyre]

It would be exciting to finally ditch some of this hex file nonsense. I'm not a big fan of 3 or 4: replacing the hexadecimal files with easy-to-edit plaintext would be better, I think, and I'm not enthusiastic about splitting them into umpteen text files. I'm not sure I quite understand 2, but 1 sounds like it might be our best option.

[hweimer]

Well, in 4 you also have an easy-to-edit JSON file in the source directory. It just gets translated to a binary file during the compilation of the game. Will probably require some CMake trickery (which is not exactly my area of expertise), but should be doable.

[peyre]

Well, however best you can do it, it would be a good thing - the main thing is to make the files easily moddable.

[hweimer]

These suggestions could be helpful.

[hweimer]

In #749, I went for option 2. Please report any issues with noticeable delays when flying a mission.

[peyre]

That's fantastic, that you've made some progress on converting a few of our binary files to plaintext. I'd love to see some of our really early Issues resolved - in addition to the whole "ease of modification" aspect.

[hweimer]

Luckily, replacing fails.cdr by a JSON version turned out to be much easier than replacing the sequence files.

[peyre]

Great to hear.

[peyre]

I've just played most of a game. The change does seem to cause slight delays following missions, but no big deal.

However, when I was getting close to launching the Moon landing, the game crashed:

WARNING /home/leon/raceintospace/src/game/fs.cpp:279:try_find_file	can't find file `PPIX_08.ogg' in audio dir(s)
unhandled exception: vector::_M_range_check: __n (which is 2954) >= this->size() (which is 2954)
Aborted (core dumped)

ppix_08.ogg does exist in raceintospace/data/video/mission/, so I'm not sure what's going on.

[hweimer]

The warning is about a missing audio file during countdown and is unrelated to the crash. 2954 is the length of the vector holding the failure sequences, so it looks like that some sequence code couldn't be found and the index variable increased beyond the bounds of the vector. I will put in some additional checks that the game doesn't crash in such a situation.

[hweimer]

In #774, I'm now going from Option 2 for deserialization (see #3 (comment)) to Option 1 as the delays from deserialization are quite significant in AI-vs-AI simulations.