bryc / mpkedit Goto Github PK

4E 54 55 45 35 31 00 05 FF 23 00 00 00 00 00 00
2D 2E 2B 28 24 0F 25 2F 25 0F 18 00 00 00 00 00
4E 54 55 45 35 31 00 05 FF 23 00 00 00 00 00 00
2D 2E 2B 28 24 0F 25 2F 25 0F 18 00 00 00 00 00

Here are two notes, they're duplicate Turok save headers that point to 0x05 on the IndexTable. The game allows it - allows you to choose from two different files, but they use the same data.

I've yet to see this occur legitimately, so it could well be that this behavior is simply not checked on the N64. Could use some investigating.

What happens when you save your game? Does it create a new file, or overwrite the current file?
Does it change the noteName? Does the game rely on the noteName in any way? Try with other games.

let are better scoped than var, etc. so except for backward compatibility it has no reason to use var rather than let

Hello, found your tool online wanting to copy a DexDrive save for use with the MiSTer FPGA N64 core now that it supports saves. However unconventional, I have uploaded the save file with the .txt extension so that you can take a look at it. The original extension is .sav.

WWF No Mercy (USA).txt

The app has gone through many design changes, and always has the goal been to break it down into individual parts. The current design (JUL 18 2015) is very simple: A self-invoking anonymous function with a bunch of functions and two variables at the root. However it is messy.

There is still more separation to do. Long nested functions to be moved down a scope. Functions that need to take more arguments. Other functionalities that need to be encapsulated. Long lines of code that need to be broken down.

One obstacle is that the root scope is function soup, and would only get messier when I do the above work. The solution would be to move these functions into modules, and expose only the methods I need. I am also going to start using JSDoc.

Everything can be delegated to one of 4 modules: file, data, ui and chromeapp

No issue just wondering something, and not like I seem able to ask over IRC. :P

Like 4 years ago when I began doing a mempak file system manager in C, I was wondering what extension to give note and page exports. I thought I had chosen ".note" and ".page" file extensions, though looking back at my C sources from 3 years ago I guess I was wrong and just used .bin instead. Instead, it appears your implementation is the one to use .note as the extension.

This just really weirds me out because I could have sworn I remembered finding some reason to use .note, or at least talking with somebody about it (may have been you), but it was too long ago to remember.

After all, .bin or maybe just using the 4-character extension code at index table node offsets header[0xC]..[0xF] would probably have worked just as well for the file extension? Or was there something special about .note?

Since you can re-order notes by dragging them, it may be a good idea to allow extracting the note by dragging it out of the browser. Right now you can do this by dragging the note save button icon out of the browser.

Update 1 (Nov28): The note sort ability is now enabled by holding CTRL. Allows dragging notes out of the browser without using the save button. However to save raw saves by dragging, they still must be saved using the save button. This is good enough for now. Only works in Chrome anyway.

Up next:

The note re-order system will have to be re-worked more.

1. A click-and-hold timer for saving is ideal, must be able to bypass normal re-order function.
2. Ability to 'revert' a re-order. The re-order should only occur when the user 'drops' or lets go of the mouse, if the mouse is off screen or not in position, revert to how it was before. Perhaps add an indicator for where the note will be moved to.

• Dragging note rows by default allow you to drag outside of the browser to save, and CTRL should correctly enable raw saves.
• Holding left mouse button while stationary for a set period of time enables sorting.
• Sorting will only take effect if the mouse is released in a valid area, if the user moves the mouse out of the page or off the table, the dragged row will revert to its original position. Dragging outside the browser should still allow saving the note.
• This should also not rely on holding CTRL.

Of course, swapping is not necessarily intuitive. You aren't moving something above another, which would require everything below to shift down by 1. You're swapping. so If you must do multiple swaps to achieve this. I'm not sure how I feel about this. On one hand, actual reordering would be the preferred way, and in reality, multiple swaps inevitably changes the data order, and in some ways it would be more scrambled than doing a traditional reorder. Perhaps in the future I can try a reorder, but for now we're keeping the simple swap method.

Filenames are used in two ways: displaying the currently opened file to the user, and when logging error or info to the console during parsing.

The filename is first available during dropHandler, and is passed into the FileReader, which is then available through fileRead. It is passed to readMemPak, and further passed to parseNoteTable.

Avoiding this constant passing of the variable would be ideal.

[1] Storing in $MPK would overwrite currently-opened file.
[2] Store as a global variable?
[3] Use closures?

Combine the backup checking into indexTable. Maybe use function recursively. Seems to work well.
Also perform index checksum repair here. Done
If implementing backup-replace, do it within parseIndexTable here. Done

When parsing data, mempak retains as much original data as possible, but sometimes performs slight modifications when necessary.

For example, the way the parsing module works, the Inode checksum is not used when checking if the filesystem is correct or not - and can be unreliable anyway. However, the Controller Pak library expects this checksum to be valid, so it has to sometimes repair broken checksums in otherwise valid files, and update it after every modification.

The NoteTable doesn't have much in terms of validation, so mempak will rely on the inode value, check if it's between 5 and 127, and also check some other offsets which are expected to be zero.

Sometimes there can be random garbage data here that, while technically won't trip up the validation, still could cause side effects, and generally uglify the file.

Perhaps all empty slots should be overwritten with zeroes, to eliminate the garbage data?

EDIT: Probably also should update the InodeTable backup after doing a modification / updating checksum.

Sorry for adding an issue, but I couldn't figure out another way of getting a hold of you.

I just wanted to say thanks for making this, and for documenting the format so carefully. I've been working on a save file converter of my own, and I wanted to support N64 DexDrive (and .mpk) files because they're quite common around the internet. I was really happy to find your documentation and implementation to look off of -- thank you!

I made mine slightly different in that if you select a cart save to pull out it automatically pads it out to the correct EEPROM size for loading in an emulator.

If you're curious, you can check out my converter here:

https://savefileconverter.com/#/n64/dexdrive
https://savefileconverter.com/#/n64/controller-pak

The code is here:

https://github.com/euan-forrester/save-file-converter

The N64 stuff is here:

https://github.com/euan-forrester/save-file-converter/tree/main/frontend/src/save-formats/N64

It also does N64 saves that are in Wii Virtual Console format, since they're also quite common around the Internet:

https://github.com/euan-forrester/save-file-converter/tree/main/frontend/src/save-formats/Wii

And I included a link to your converter:

https://savefileconverter.com/#/other-converters

Anyway, none of this would have been possible without your work figuring out and documenting the format, so thank you again!

BTW, you mentioned that there are some bytes in the N64 DexDrive format that you couldn't figure out. I noticed it's quite similar to the PS1 DexDrive format, and in that one some of the bytes are apparently copies of the file table and linked list structures in the actual memcard file:

https://github.com/euan-forrester/save-file-converter/blob/main/frontend/src/save-formats/PS1/DexDrive.js#L76

(The memcardrex file converter that I cite there can actually talk to a real PS1 DexDrive!)

No idea if it's the same in the N64 case, but I wanted to mention it anyway.

Thank you again!

    var Parser = {
            "error": {
                "types":[],
                "count": 0
            },
            "indexes": [],
            "noteCount": 0
    }

I copy a template like the above to both the NoteTable Parser and the IndexTable Parser functions. It would be nice to make it more streamlined, where its only instantiated once, like a prototype or constructor type thing.

Edit: This is all obsolete stuff. There is no need to count number of errors and review it later. Error checking is done during parsing and can cease parsing if the error is bad enough. In the future, debugging information may be collected in a similar object and displayed in a debug menu.

Implement noteTable wipe in noteTableParse. Wipe all non-valid sections for cleaner files. Automatically re-order/move entries up when deleting notes?
Also perform a wipe to the unused header areas.

In your code, sometimes you use template literals, sometimes you used string concatenation. It's better to use template literals when working with variables, as it's more readable.

Guessing it's not really an "issue", in general.

Not sure how often this has been tested in those Mozilla-family-based browsers, but, based on the error console in one of those, I always get these six warnings every time I load the page via rawgit.com:

Warning: Unknown property '-moz-osx-font-smoothing'.  Declaration dropped.
Source File: http://rawgit.com/bryc/mempak/master/fonts/font-awesome.min.css
Line: 4, Column: 311

Warning: Expected 'none' or URL but found 'progid'.  Error in parsing value for 'filter'.  Declaration dropped.
Source File: http://rawgit.com/bryc/mempak/master/fonts/font-awesome.min.css
Line: 4, Column: 1454

Warning: Expected 'none' or URL but found 'progid'.  Error in parsing value for 'filter'.  Declaration dropped.
Source File: http://rawgit.com/bryc/mempak/master/fonts/font-awesome.min.css
Line: 4, Column: 1617

Warning: Expected 'none' or URL but found 'progid'.  Error in parsing value for 'filter'.  Declaration dropped.
Source File: http://rawgit.com/bryc/mempak/master/fonts/font-awesome.min.css
Line: 4, Column: 1783

Warning: Expected 'none' or URL but found 'progid'.  Error in parsing value for 'filter'.  Declaration dropped.
Source File: http://rawgit.com/bryc/mempak/master/fonts/font-awesome.min.css
Line: 4, Column: 1954

Warning: Expected 'none' or URL but found 'progid'.  Error in parsing value for 'filter'.  Declaration dropped.
Source File: http://rawgit.com/bryc/mempak/master/fonts/font-awesome.min.css
Line: 4, Column: 2127

parseNoteTable:

1. gameCode typically must exist, and cannot be 00 00 00 00. Most games rely on this for identifying associated notes. For example, Gex 64 (NX2E) relies on the noteName only (In the format GEX:%s), and any gameCode will function as long as it's not completely empty.
2. publisherCode also typically must exist, and cannot be 00 00 although some games don't care.
   3, noteName is typically not empty, and many games rely on this, especially when making use of extensions.
   Perhaps use a more strict mode by default, and have an option for a more lax mode (for salvaging repairable data).

headerCheck:

1. Certain bits in the data must be set to avoid some errors (Pak is full, etc). However, headerCheck validates checksums without setting it. These bits should be fixed if they are not set, to be more robust.

See wiki entry