bryc / mpkedit

MPKEdit - Nintendo 64 (N64) Controller Pak manager in JavaScript

MPKEdit - Controller Pak Manager

Makes managing N64 Controller Pak files (.MPK) easy, whether from an emulator or a flash cart. MPKEdit is written in JavaScript and works in most modern browsers (Chrome, Firefox, Opera, etc.).

Try it online

Features

• Open .MPK and .note files by dragging and dropping them, or browsing for files
• Save, load, delete, and re-order individual save files (aka "notes")
• Add comments to your save files
• DexDrive (.n64) import support
• Game code database - automatically identifies game name and publisher
• Inspect metadata from the MPK file and other details

Instructions

To open files, click the file icon at the top left, or drag and drop into the window. Save the file by clicking the Save (floppy) icon at the top right. Individual save files (notes) can be saved by clicking the save icon at the rightmost of each row. It is also possible to re-order notes by clicking and dragging with reorder mode enabled in Settings.

You can also add text comments to any of the 16 save slots, accessible by clicking the info button beside the save icon. When a note has a comment assigned, a text bubble appears for that entry, which displays the comment when hovered. These will be saved in both .note or .MPK files. Be careful when using .MPKs with comments in an emulator as they may not know how to handle the appended data and it may be lost. Always keep it backed up somewhere.

Basic Mupen64Plus-Next Save Support

The main N64 emulator in RetroArch is Mupen64Plus-Next, and it bundles all possible save data into one large 290 KB .SRM file per game. MPKEdit has basic support for loading and saving these files, but there are a few limitations. Only the data of Port 1's Controller Pak is loaded. If a game uses EEPROM/SRAM as well, those will not be kept when saving. Controller ports 2, 3 and 4 are ignored. Believe it or not, some games (BattleTanx) let you save to any port, even if you're using port 1, so try not to do that. But for the most part, most games don't use both Controller Pak and internal saves, so it should be fine. This might be improved in the future, but likely would require significant code changes, which aren't worth it just for this.

Advanced

For advanced users, holding the Control key while saving a note activates raw note saving (when activated, save icons turn red). This saves only the raw save data itself without any headers. This is intended for situations where the header can be ignored. It is possible to reimport the modified .rawnote file by loading it back into the original .MPK file that it came from. You can also add description text to the end of the filename, Just don't change the initial "raw-XXX_XX" part at the beginning of the filename, as that is how it knows where to put it back. The modified .rawnote data MUST be the same size as the original, as you cannot alter the size of the note data this way.

Mempaks are sometimes used with utilities like GameShark to back up cartridge saves (.EEP files), so the .rawnote files will work in emulators once renamed properly. Currently, GameShark features are not officially supported, it's just a handy tip. BOBdotEXE made a nice tutorial covering more advanced GameShark usage here, but a hex editor will be required: https://www.youtube.com/watch?v=PpolokImIeU.

In Chrome, it is possible to save the MPK or notes to a specific folder by clicking and dragging the save button to your destination or folder. This may not work in all environments.

As for the blocky icons, those are just a way to identify unique save data, useful if you have multiple saves of the same game and need to differentiate between them. It is disabled by default but can be enabled in the Options menu. It ended up turning into a fun little side project I'm still working on. Even the smallest change will produce a radically different icon. Useful for finding duplicates too. It's disabled by default, but you can enable them if you want to see what your save's icons look like or just think they're pretty. There should be trillions of different icons possible, so each one is unique!

mpkedit's Issues

Replace var by let

let are better scoped than var, etc. so except for backward compatibility it has no reason to use var rather than let

Move everything to parseIndexTable

Combine the backup checking into indexTable. Maybe use function recursively. Seems to work well.
Also perform index checksum repair here. Done
If implementing backup-replace, do it within parseIndexTable here. Done

Thank you for such an awesome editor!

Sorry for adding an issue, but I couldn't figure out another way of getting a hold of you.

I just wanted to say thanks for making this, and for documenting the format so carefully. I've been working on a save file converter of my own, and I wanted to support N64 DexDrive (and .mpk) files because they're quite common around the internet. I was really happy to find your documentation and implementation to look off of -- thank you!

I made mine slightly different in that if you select a cart save to pull out it automatically pads it out to the correct EEPROM size for loading in an emulator.

If you're curious, you can check out my converter here:

https://savefileconverter.com/#/n64/dexdrive
https://savefileconverter.com/#/n64/controller-pak

The code is here:

https://github.com/euan-forrester/save-file-converter

The N64 stuff is here:

https://github.com/euan-forrester/save-file-converter/tree/main/frontend/src/save-formats/N64

It also does N64 saves that are in Wii Virtual Console format, since they're also quite common around the Internet:

https://github.com/euan-forrester/save-file-converter/tree/main/frontend/src/save-formats/Wii

And I included a link to your converter:

https://savefileconverter.com/#/other-converters

Anyway, none of this would have been possible without your work figuring out and documenting the format, so thank you again!

BTW, you mentioned that there are some bytes in the N64 DexDrive format that you couldn't figure out. I noticed it's quite similar to the PS1 DexDrive format, and in that one some of the bytes are apparently copies of the file table and linked list structures in the actual memcard file:

https://github.com/euan-forrester/save-file-converter/blob/main/frontend/src/save-formats/PS1/DexDrive.js#L76

(The memcardrex file converter that I cite there can actually talk to a real PS1 DexDrive!)

No idea if it's the same in the N64 case, but I wanted to mention it anyway.

Thank you again!

Improved note dragging/dropping for saving and sorting

Since you can re-order notes by dragging them, it may be a good idea to allow extracting the note by dragging it out of the browser. Right now you can do this by dragging the note save button icon out of the browser.

Update 1 (Nov28): The note sort ability is now enabled by holding CTRL. Allows dragging notes out of the browser without using the save button. However to save raw saves by dragging, they still must be saved using the save button. This is good enough for now. Only works in Chrome anyway.

Up next:

The note re-order system will have to be re-worked more.

1. A click-and-hold timer for saving is ideal, must be able to bypass normal re-order function.
2. Ability to 'revert' a re-order. The re-order should only occur when the user 'drops' or lets go of the mouse, if the mouse is off screen or not in position, revert to how it was before. Perhaps add an indicator for where the note will be moved to.

• Dragging note rows by default allow you to drag outside of the browser to save, and CTRL should correctly enable raw saves.
• Holding left mouse button while stationary for a set period of time enables sorting.
• Sorting will only take effect if the mouse is released in a valid area, if the user moves the mouse out of the page or off the table, the dragged row will revert to its original position. Dragging outside the browser should still allow saving the note.
• This should also not rely on holding CTRL.

Of course, swapping is not necessarily intuitive. You aren't moving something above another, which would require everything below to shift down by 1. You're swapping. so If you must do multiple swaps to achieve this. I'm not sure how I feel about this. On one hand, actual reordering would be the preferred way, and in reality, multiple swaps inevitably changes the data order, and in some ways it would be more scrambled than doing a traditional reorder. Perhaps in the future I can try a reorder, but for now we're keeping the simple swap method.

dropHandler: Global filename access

Filenames are used in two ways: displaying the currently opened file to the user, and when logging error or info to the console during parsing.

The filename is first available during dropHandler, and is passed into the FileReader, which is then available through fileRead. It is passed to readMemPak, and further passed to parseNoteTable.

Avoiding this constant passing of the variable would be ideal.

[1] Storing in $MPK would overwrite currently-opened file.
[2] Store as a global variable?
[3] Use closures?

Making the parser more closely resemble the N64 libs

parseNoteTable:

1. gameCode typically must exist, and cannot be 00 00 00 00. Most games rely on this for identifying associated notes. For example, Gex 64 (NX2E) relies on the noteName only (In the format GEX:%s), and any gameCode will function as long as it's not completely empty.
2. publisherCode also typically must exist, and cannot be 00 00 although some games don't care.
   3, noteName is typically not empty, and many games rely on this, especially when making use of extensions.
   Perhaps use a more strict mode by default, and have an option for a more lax mode (for salvaging repairable data).

headerCheck:

1. Certain bits in the data must be set to avoid some errors (Pak is full, etc). However, headerCheck validates checksums without setting it. These bits should be fixed if they are not set, to be more robust.

See wiki entry

Miscellaneous things

Implement noteTable wipe in noteTableParse. Wipe all non-valid sections for cleaner files. Automatically re-order/move entries up when deleting notes?
Also perform a wipe to the unused header areas.

Parser: Central object or constructor for Parser?

    var Parser = {
            "error": {
                "types":[],
                "count": 0
            },
            "indexes": [],
            "noteCount": 0
    }

I copy a template like the above to both the NoteTable Parser and the IndexTable Parser functions. It would be nice to make it more streamlined, where its only instantiated once, like a prototype or constructor type thing.

Edit: This is all obsolete stuff. There is no need to count number of errors and review it later. Error checking is done during parsing and can cease parsing if the error is bad enough. In the future, debugging information may be collected in a similar object and displayed in a debug menu.

Controller Pak allows notes using the same Index?

4E 54 55 45 35 31 00 05 FF 23 00 00 00 00 00 00
2D 2E 2B 28 24 0F 25 2F 25 0F 18 00 00 00 00 00
4E 54 55 45 35 31 00 05 FF 23 00 00 00 00 00 00
2D 2E 2B 28 24 0F 25 2F 25 0F 18 00 00 00 00 00

Here are two notes, they're duplicate Turok save headers that point to 0x05 on the IndexTable. The game allows it - allows you to choose from two different files, but they use the same data.

I've yet to see this occur legitimately, so it could well be that this behavior is simply not checked on the N64. Could use some investigating.

What happens when you save your game? Does it create a new file, or overwrite the current file?
Does it change the noteName? Does the game rely on the noteName in any way? Try with other games.

Code design change: break the app down into modules

The app has gone through many design changes, and always has the goal been to break it down into individual parts. The current design (JUL 18 2015) is very simple: A self-invoking anonymous function with a bunch of functions and two variables at the root. However it is messy.

There is still more separation to do. Long nested functions to be moved down a scope. Functions that need to take more arguments. Other functionalities that need to be encapsulated. Long lines of code that need to be broken down.

One obstacle is that the root scope is function soup, and would only get messier when I do the above work. The solution would be to move these functions into modules, and expose only the methods I need. I am also going to start using JSDoc.

Everything can be delegated to one of 4 modules: file, data, ui and chromeapp

Just a question about .note extension.

No issue just wondering something, and not like I seem able to ask over IRC. :P

Like 4 years ago when I began doing a mempak file system manager in C, I was wondering what extension to give note and page exports. I thought I had chosen ".note" and ".page" file extensions, though looking back at my C sources from 3 years ago I guess I was wrong and just used .bin instead. Instead, it appears your implementation is the one to use .note as the extension.

This just really weirds me out because I could have sworn I remembered finding some reason to use .note, or at least talking with somebody about it (may have been you), but it was too long ago to remember.

After all, .bin or maybe just using the 4-character extension code at index table node offsets header[0xC]..[0xF] would probably have worked just as well for the file extension? Or was there something special about .note?

NoteTable: Wipe the NoteTable clean

When parsing data, mempak retains as much original data as possible, but sometimes performs slight modifications when necessary.

For example, the way the parsing module works, the Inode checksum is not used when checking if the filesystem is correct or not - and can be unreliable anyway. However, the Controller Pak library expects this checksum to be valid, so it has to sometimes repair broken checksums in otherwise valid files, and update it after every modification.

The NoteTable doesn't have much in terms of validation, so mempak will rely on the inode value, check if it's between 5 and 127, and also check some other offsets which are expected to be zero.

Sometimes there can be random garbage data here that, while technically won't trip up the validation, still could cause side effects, and generally uglify the file.

Perhaps all empty slots should be overwritten with zeroes, to eliminate the garbage data?

EDIT: Probably also should update the InodeTable backup after doing a modification / updating checksum.

importNote: Error message when no notes left

Error console gets spammed with CSS warnings.

Guessing it's not really an "issue", in general.

Not sure how often this has been tested in those Mozilla-family-based browsers, but, based on the error console in one of those, I always get these six warnings every time I load the page via rawgit.com:

Warning: Unknown property '-moz-osx-font-smoothing'.  Declaration dropped.
Source File: http://rawgit.com/bryc/mempak/master/fonts/font-awesome.min.css
Line: 4, Column: 311

Warning: Expected 'none' or URL but found 'progid'.  Error in parsing value for 'filter'.  Declaration dropped.
Source File: http://rawgit.com/bryc/mempak/master/fonts/font-awesome.min.css
Line: 4, Column: 1454

Warning: Expected 'none' or URL but found 'progid'.  Error in parsing value for 'filter'.  Declaration dropped.
Source File: http://rawgit.com/bryc/mempak/master/fonts/font-awesome.min.css
Line: 4, Column: 1617

Warning: Expected 'none' or URL but found 'progid'.  Error in parsing value for 'filter'.  Declaration dropped.
Source File: http://rawgit.com/bryc/mempak/master/fonts/font-awesome.min.css
Line: 4, Column: 1783

Warning: Expected 'none' or URL but found 'progid'.  Error in parsing value for 'filter'.  Declaration dropped.
Source File: http://rawgit.com/bryc/mempak/master/fonts/font-awesome.min.css
Line: 4, Column: 1954

Warning: Expected 'none' or URL but found 'progid'.  Error in parsing value for 'filter'.  Declaration dropped.
Source File: http://rawgit.com/bryc/mempak/master/fonts/font-awesome.min.css
Line: 4, Column: 2127

Replace string concatenation by template literals

In your code, sometimes you use template literals, sometimes you used string concatenation. It's better to use template literals when working with variables, as it's more readable.

Support for MiSTer N64 saves?

Hello, found your tool online wanting to copy a DexDrive save for use with the MiSTer FPGA N64 core now that it supports saves. However unconventional, I have uploaded the save file with the .txt extension so that you can take a look at it. The original extension is .sav.

WWF No Mercy (USA).txt