Support for MiSTer N64 saves? about mpkedit

[euan-forrester]

I'm not the owner of this repo, but I'll be adding support for this to https://savefileconverter.com/#/mister sometime soon (just gotta get through moving homes first :) ). You can checkout the github repo here: https://github.com/euan-forrester/save-file-converter

For reference to the repo owner here, should they also wish to add support for this, the format is:

• Optional cart save of whatever size (0.5kB - 128kB), then
• Optional 4 controller pak saves, in the usual raw .mpk format

So you can generally tell what's in the file just by looking at the size. The exception is a Flash RAM save is the same size as 4 controller pak saves. So to disambiguate there you could try and parse the file as controller pak saves.

The wrinkle with that is that the controller pak data defaults to just all 0x00, which is not a valid controller pak save but is valid to the MiSTer to indicate uninitialized data. So in the case that the file is 128kB in size and all 0x00 then it could be either a blank Flash RAM save or 4 blank controller pak saves.

EDIT: Also, the cart save is in emulator byte order and not original hardware byte order (I can never remember which is big and which is little).

[KingKannibal]

Here is what the split saves look like in the current test build of MiSTer Main where they are experimenting with split saves.
N64_20231125 Split Saves.zip

[bryc]

I'll look into it. I already have partial support for Mupen64plus-Next saves which do a similar internal "all-in-one" bundling. Though the way it currently works is that I just straight up zero-fill everything except the P1 Controller Pak data. As it was simpler to implement that way. I figured that most games that use Controller Pak don't use any other save type. WWF No Mercy is one game that happens to use both. So I suppose it's also time for a more rigorous solution.

The way I'll probably go about doing it, is to first detect the .sav extension, Then try to detect any valid Controller Pak data at specific offsets using this table (based on @euan-forrester's explaination, I don't have a MiSTer so I have no clue if it's right 😄 ):

Save Type	Structure	Final Size	Example Game
4K EEPROM	512 + (32768 * 4)	131584	Super Mario 64
16K EEPROM	2048 + (32768 * 4)	133120	Donkey Kong 64
256K SRAM	32768 + (32768 * 4)	163840	Zelda 64
1M FLASH	131072 + (32768 * 4)	262144	WWF No Mercy

EDIT: Also, the cart save is in emulator byte order and not original hardware byte order (I can never remember which is big and which is little).

That sounds weird. You sure? It should really just match N64 hardware, which is big-endian. Though PC emulators do seem to leave EEPROM and Controller Pak data in their original form, it's just really SRAM or Flash that seems to have the swap occur. Like Harvest Moon was one game where an Everdrive save had to be swapped to work in an emulator; but EEPROM and MPK files did not.

The wrinkle with that is that the controller pak data defaults to just all 0x00, which is not a valid controller pak save but is valid to the MiSTer to indicate uninitialized data.

I'm a bit confused. I'm looking at the MiSTer code and it seems to use actual template data from Mupen64, not 0x00's?

https://github.com/RobertPeip/Mister64/blob/5c163b73e41591dabf68e41617b89d4c16baf150/rtl/pif_cpakinit.vhd#L19

[euan-forrester]

I think that the overall caveat with this n64 mister stuff is that it's all a work-in-progress and subject to change. I was warned to not implement this stuff yet for the savefileconverter site because the core is not stable, but it's quickly becoming popular and I've had a few users now asking about it so I went ahead and began.

That sounds weird. You sure? It should really just match N64 hardware, which is big-endian.

Yeah I was surprised too. I tested everything carefully though. I got a directory of sample files from a user, and went through them to find a save of each type with actual gameplay data in it so I could verify it loads correctly.

If you want to check out some sample data, you can take a look at my work-in-progress branch: https://github.com/euan-forrester/save-file-converter/tree/n64-mister

Tests: https://github.com/euan-forrester/save-file-converter/blob/n64-mister/frontend/tests/unit/save-formats/Mister/N64.spec.js

Data: https://github.com/euan-forrester/save-file-converter/tree/n64-mister/frontend/tests/data/save-formats/mister/n64

The wrinkle with that is that the controller pak data defaults to just all 0x00, which is not a valid controller pak save but is valid to the MiSTer to indicate uninitialized data.

I'm a bit confused. I'm looking at the MiSTer code and it seems to use actual template data from Mupen64, not 0x00's?

https://github.com/RobertPeip/Mister64/blob/5c163b73e41591dabf68e41617b89d4c16baf150/rtl/pif_cpakinit.vhd#L19

I'm not sure what to say here. The example files I was given had all 0x00 for all the controller pak data. I haven't even found a file with valid controller pak data in it yet, but they're the same size as raw .mpk files and the mister for the most part just uses raw data. I see the commit that made the change you linked was merged about 2 months ago. I don't know the mister release process, but it seems that (at least in these early times) we can expect users to have saves from earlier versions of the core.

[KingKannibal]

Thanks, it's great to see you guys collaborating :D

[bryc]

It sounds like the current .sav file format is just a quick fix not set in stone, and will be subject to change. It sounds like the final core might even have separated save files, and so, special support wouldn't need to be added if that happens.

That said, I will still fix Mupen64Plus-next saves , since lack of dual save support in WWF No Mercy is a big oversight. And I should also be able to do Mister64 beta save support at least with minimal code.

@KingKannibal Would you be able to make a few more save files for the various save types? Basically make save files that contain both primary saves and controller pak saves? Would give me files to test with.

Some easy ones:
Perfect Dark - EEPROM 16K + Controller Pak (easy as it creates C-Pak data instantly, was there a bypass patch for that one?)
Diddy Kong Racing - EEPROM 16K + Controller Pak
Mario Kart 64 - EEPROM 4K + Controller Pak (when saving a ghost)
WWF WrestleMania 2000 - SRAM + Controller Pak (probably easy for you as you're familiar with these games)
Banjo Kazooie - EEPROM 4K + Controller Pak (little known fact - BK will actually read and attempt a repair during boot; so extra data should be saved to the file!)
Banjo Tooie - EEPROM 16K + Controller Pak (little known fact - BT will actually read and attempt a repair during boot; so extra data should be saved to the file!)

There's actually not that many games that support both, but those ones are probably the easiest to make a save on.

[KingKannibal]

Sure, I'll have these out as soon as possible.

[KingKannibal]

Process for each game:

[Let me know if any of these need to be redone]

• Diddy Kong Racing: Saved a ghost
• Mario Kart 64: Saved a ghost
• WWF WrestleMania 2000: Saved Superstar and Belt data
• Banjo Kazooie: I was not prompted with any repair screen, but started a game then saved and quit.

Games that don't boot yet as of N64_20231115

• Perfect Dark [Requires a high performance patch. I will be providing a file from this version.]
• Banjo-Tooie [Requires an NTSC Crack to bypadd anti-piracy. I will be providing a file from this version.]

edit: I didn't know I could upload zips
N64 Saves [N64_20231115].zip

[euan-forrester]

@KingKannibal Could you please re-create at least one of these (e.g. Mario Kart 64, since it's easy to use for testing) with the latest version of the core?

The controller pak data is quite corrupted, and fails several integrity checks that I'd rather not disable. bryc helped the developer of the core fix some issues related to this, and I'm hopeful that newer versions of the core won't have this issue.

BTW, the controller pak data needs to be endian swapped before it's readable, which was another endian-related surprise with this data.

[KingKannibal]

@KingKannibal Could you please re-create at least one of these (e.g. Mario Kart 64, since it's easy to use for testing) with the latest version of the core?

The controller pak data is quite corrupted, and fails several integrity checks that I'd rather not disable. bryc helped the developer of the core fix some issues related to this, and I'm hopeful that newer versions of the core won't have this issue.

BTW, the controller pak data needs to be endian swapped before it's readable, which was another endian-related surprise with this data.

Yes, I'll be supplying new saves asap.

[KingKannibal]

@euan-forrester Here's the save file made with the latest core: N64_20231124
Mario Kart 64 (USA).zip

[euan-forrester]

@KingKannibal Wow, thanks for the quick turnaround!

It looks like you didn't make any gameplay progress in this save? Just started the game and then took the save file?

It's good to see that, because now it looks like the core correctly initializes all 4 controller paks rather than leaving them at all 0x00 which is a big improvement.

Could I ask you to do another Mario Kart save with this core -- maybe do a time trial so that there's data saved in the cartridge area, and also save a ghost so there's data in the controller pak as well?

Then I think I'm good for files! Thanks to you for all this testing, and thanks to @bryc for helping to improve the core so that the files are better!

[KingKannibal]

@euan-forrester @bryc
I have two new bits of info for you guys.

1. new core is out and I'll send a clean save file along with a save file that includes ghost data for Mario Kart 64.
2. It looks like they are experimenting with splitting the saves (cartridge save and individual pak saves) so this changes a lot if they decide to implement it into a stable release of MiSTer Main.

[KingKannibal]

Here are the save files produced with the latest core [as of posting]. Includes a clean save, and a save with ghost data.
N64_20231125 Saves.zip

[euan-forrester]

@KingKannibal Thanks for all of that! And for noticing that there's a new experiment with split saves (I'm not sure where you saw that!)

I've finished my conversion code, and incorporated your saves into my automated tests for the 'current' format, but before writing the UI I think it makes sense to pause here and see what direction the mister team takes with their save format, and wait for the format to become more finalized.

I was warned earlier not to work on this yet, but after a few users asked for it I wanted to proceed anyway. But it seems that yeah it's too early.

It's not really clear to me how to determine when the save format is (close enough to) finalized, other than waiting until the core moves under https://github.com/MiSTer-devel

[KingKannibal]

@euan-forrester Yeah, I'm gonna wait to see if the end up switching to the split saves or staying under the singular format.

[bryc]

Looks good. If the split saves method ends up practical and reliable, that'd be the way to go IMO. It loads in MPKEdit directly just fine.

Let me know if they go forward with split saves or decide to go back to single files :)

[KingKannibal]

Sure thing, I'll try my best to keep up with developments regarding saves.