travisgoodspeed / maskromtool Goto Github PK

Release v2023-08-06 is broken on Windows for missing DLLs, likely as a recent change of the Windows build script. v2023-07-20 is unaffected and works fine.

Gatorom is a CLI tool and C++ library that I've written for decoding bitstreams into logical bytes, something of a friendly competitor to Zorrom. We'll use this issue to track its inclusion into MaskRomTool's git repository and binary distributions.

• Code
• Documentation
• Tests
• Binaries on macOS.
• Binaries on Windows.
• Binaries on Linux.
• Library integration to MRT.

The alignment strategy implemented in RomAlignerDefault is accurate, but it takes many seconds on large project files. Close this issue when faster strategies are available for selection and a test case to compare the alignment of a large project between two strategies.

Sean Riddle has an excellent collection of decapsulated ROMs from old video games, but he shares the dumps as binary files rather than the ASCII art that others have standardized on. Use this issue to track support for his format.

Some details and a Python decoder can be found in Zorrom.

Running cmake --install or make install does not perform any actions due to a missing install section in CMakeLists.txt.

Describe the bug
The alignment algorithm fails when there are a very small number of bits marked on the screen.

To Reproduce
Steps to reproduce the behavior:

1. Mark the first eight columns and eight rows of any ROM.
2. Export to ASCII.
3. The export will not show the proper number of rows and columns.

Expected behavior
We expect that eight columns and eight rows will be identified, instead of the mismash that are. Larger ROMs are not affected.

Screenshots

Statistical analysis of the pixel values might be interesting. Close this issue when we can export the bit RGB values.

Running in Wayland mode makes the program very unstable, and users are forced to fall back to X11 mode by passing -platform xcb on the CLI. You can repeat the crash by opening a large project file and holding the Tab button to rapidly show and hide all of the bits.

Close this issue when Wayland is stable.

A performance test recently showed that 73% of time in a tight markBits() loop was spent in QGraphicsScenePrivate::removeItemHelper. We could therefore dramatically improve performance by only removing those bits on a line when moving it, rather than regenerating all bits.

Similarly, when we move a line, we can simply cut all of its bits and then recreate them in the new position. This will be much faster than removing all bits to regenerate them.

Marking a row or column just a little bit off with make the bits unreadable, and users often need to translate the position of the object just a little bit. Currently this is done by using the S key to Set the position to where the mouse is, but that can be convenient for very long lines or when the far end of the line is the anchor point.

Close this issue when three new methods of translation are available:

• Three fingered drag on a trackpad.
• Right-click drag with a mouse.
• Keyboard steps.

In offscreen mode, the primary and secondary views cannot enable OpenGL. This doesn't break anything, but the warning messages are a little distracting. We can fix this by skipping the OpenGL enablement if the platform is offscreen.

air% maskromtool msp430f149.bmp -a msp430f149.txt -platform offscreen -e -d
QOpenGLWidget is not supported on this platform.
QOpenGLWidget is not supported on this platform.
This plugin does not support createPlatformOpenGLContext!
This plugin does not support createPlatformOpenGLContext!
QOpenGLWidget: Failed to create context
Exporting to ASCII.
air% 

An operator can sometimes get lost within a large project file, and it would be handy to have a small view on the side of the whole project, so that high level navigation can occur without adjusting the zoom.

• Second view is visible.
• Keyboard presses apply to both.
• Crosshairs extend through both views, not just the active one.

Qt contains a "best effort" model for adjusting the internal font when none is available in the GUI. This way if we don't have a font, a similar font will be selected, and no one but a graphic designer ought to care. This works well enough in GUI mode.

When running in CLI mode, many users add -platform offscreen -e to prevent any windows from opening and to exit when the program is done. On macOS, this has the unintended side effect of preventing the font substitution code in Qt from caching the replacement font, resulting in warning messages like this one.

qt.qpa.fonts: Populating font family aliases took 444 ms. Replace uses of
missing font family "Sans Serif" with one that exists to avoid this cost. 

The fix is simple: Just define a macOS font if none exists in offline mode.

Close this issue when we a package an installer using the QT Installer Framework.

• Linux
• Windows
• macOS

Running the application on Windows requires the Visual Studio redistributable package. Without it, the following error will pop up when the app has been built with Visual Studio 2022 and installed on a brand new machine. This is difficult to reproduce on a development machine because, of course, that machine will already have its own Visual Studio installed.

We can fix this by either:

1. Linking to the latest redistributable package; or,
2. Including concrt140.dll, msvcp140.dll, vccorlib140.dll, and vcruntime140.dll and perhaps some others when buildrelease.bat builds the package.

Performance is decently fast, but it can be a little frustrating in projects of hundreds of thousands of bits. A performance test would allow us to judge algorithmic improvements and scale to millions of bits.

Linear bit histograms work great for ROMs, in which ones and zeroes occur with roughly equal quantity, but it's a rather poor way to view HAL logic arrays, in which ones are rather rare and zeroes are plentiful. Close this ticket when we have a checkbox to view the histogram on a log scale, where the difference is less troublesome.

The default color scheme works will on brightfield microscopy but is difficult to see in darkfield. Close this issue when the colors are adjustable.

Many projects have identical slopes for every row or column. While we can't choose the best slope when reopening, in these files it might suffice to re-use the first row or column's tilt as the default for all other lines.

It would be nice to add text annotations of a selectable color, so that features can be explained within the project rather than in scattered screenshots.

The current code fails when there are large vertical gaps between bits, because it inaccurately counts the rows.

It's currently difficult to tell which item is selected before operating on it. Close this issue when we have a visual indicator, such as a highlight box or changing the color of the lines.

On Windows, a PE file includes a flag to describe whether it is a console or a GUI application. Close this issue when our CMakefile has been modified to produce two executables on windows, one that blocks for the CLI and one that launches as a GUI application.

I screwed up decoding the MYK82 ROM for about a week because I was only reading half the words on each row. Close this issue when we have this added to the status line, so I don't make such mistakes in the future.

The current status line is adequate, but it would be nice to include some more information.

Close this issue when we have a convenient setting to choose whether the bits are inverted. Specifically, I need this for some photos of the MYK82.

The current alignment algorithm, RomAlignerNew works great when there's a little tilt to the image and overlap between rows is limited to a few bits in a row when sorted by the X axis. Images with no tilt at all, and images with more overlap, require that a threshold be set to calculate the appropriate number of bits that might arrive out of order in determining the first column.

Close this issue when we have a new alignment algorithm that does not require such fine tuning and works on all existing samples.

Our wide sampler is quite effective on diffusion ROMs that need it. Close this issue when we have a tall sampler as well.

The tool will crash if you first delete a line and then hit V to run the DRC rules. This was probably introduced with performance optimizations.

We can fix this by running markBits() before the DRC rules. That's probably more stable than carefully maintaining the the dirty status of the alignment.

NEC's uCOM4 chips use a custom format which is much like cols-downl except that every other pair of pages is swapped. Close this issue when it has been implemented and we have a passing test case for Fabulous Fred.

• Fabulous Fred Decodes
• uCOM4 binaries are solved.
• Module is cleaned and parameterized.

We currently have the beginnings of support for OpenGL, but it is not yet useful. Close this issue when we've fixed the following OpenGL bugs:

• Large background images become black in OpenGL.
• The crosshairs cease to move in OpenGL mode.
• Zooming out makes sections of row and column lines become invisible in OpenGL mode.

Users launching the GUI directly would like to be able to re-open the most recent project file. Close this issue when that exists in the File menu.

After annotating a ROM, it's nice to know that the bits are correct. One way to do that is to have two projects, and to compare them, correcting disagreeing bits until all of them match.

Close this issue when we support that in the GUI and in the CLI.

Now that libraries have been integrated with #42, let's use this issue to track adjustments to integrate it into the GUI.

• Settings for GatoROM maintained in .json.
• Search GUI.
• Hex GUI.
• Visualizing bits in the main display.

Close this issue when:

• Pressing H will jump to a home location.
• Pressing Shift+H will assign a new home location.

Currently the SKIPCOUNTTHRESHOLD definition in romalignernew.cpp defines how many pixels can be found out of order before the first column. It is also a source of confusing bugs, because if the first two columns do not overlap, but instead form a \/ or /\, there will be no skipped bits between real bits and the algorithm might be confused.

Close this issue when the threshold is selectable. 5 should be a decent default.

Shown in the image below, windows sometimes tears when panning under Microsoft Windows. The image is restored by resetting the zoom on the keyboard.

First observed on this Intel NUC.

Processor	11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz   2.42 GHz
Installed RAM	16.0 GB (15.7 GB usable)
Device ID	B6C0D57E-B14A-4A46-94AD-519A2671BE3A
Product ID	00330-54283-81039-AAOEM
System type	64-bit operating system, x64-based processor
Pen and touch	No pen or touch input is available for this display
Edition	Windows 11 Pro
Version	22H2
Installed on	‎2023-‎07-‎18
OS build	22621.1992
Experience	Windows Feature Experience Pack 1000.22644.1000.0

We need to include the README file in the installer, to alleviate user confusion.

Currently the bits are marked by the intersection of row and column lines. It would reduce the operator's labor to be able to draw grids, marking many evenly spaced rows and columns in a single burst.

Note that the grid will not exclude row and column markings. They may be mixed.

We currently sample just one pixel for each bit. This is surprisingly effective but some, particularly diffusion ROMs under good focus and magnification, require other sampling methods.

Close this issue when we can adjust the sample size, perhaps with some choices as to how the colors of the sample spot are merged.

The GatoROM class contains a description() method that returns a QString of the configured parameters, suitable for use on the CLI. What we don't yet have, but sorely need, is a method to configure a GatoROM instance from such a string, to facility loading and saving in the GUI.

A release copy for Windows would be handy, as many Windows users aren't comfortable compiling C++ code. Use this issue to track its builds and close when it's decently repeatable.

For small images, it's very convenient to redraw in the GUI thread. For larger images, this creates a lag that can become infuriating, as pending GUI updates queue behind one another rather than skipping the steps that will soon be deleted.

Moving updates into their own thread could dramatically improve the responsiveness of the GUI, if care is taken not to break the CLI. Maybe a unix-ish sync() function could write all pending updates, only as needed?

Very large files, those over one gigabyte, fail to load. That's more than we need for many ROMs, but will become inconvenient at some point.

Describe the bug
I can run your Mask ROM Tool by compiling it through QT Creator.
However, when clicking File -> Open ROM I cannot select any .bmp image. It does not show the file in the file explorer.
You have to type the file name in manually to load up the image.

To Reproduce

1. Clicking File -> Open ROM

Expected behavior
You should be able to select/see the files in the file explorer

Screenshots

Desktop (please complete the following information):

• Windows 10

Absolute thresholds usually work well for Diffusion and Via ROMs, but sometimes focal drift or the chemical etching drift of an Implant ROM means that one region of the image will decode better than others.

Close this issue when the program supports comparing a bit to a rolling average of recent bits, matching it to either a one or a zero by relative rather than absolute thresholds.

When GatoROM can't solve a ROM, it would be handy to print the bits to paper for manual exploration. Close this ticket when we have QT Printing support for printing the ASCII art to a single page.

Printing an image might also be handy, but that's not tracked by this issue.

It would be nice to have built in support for decoding the MYK82 ROM.

Here's a rough python script.

#!/usr/bin/env python3

from rom10 import bits;


### This is a partial decoder by Travis Goodspeed for the MYK82 ROM.
### As things stand, rows of 16 words (512 bits) are properly decoded.

### Output is a nasm source file, describing the row numbers and 32-bit words.

lastrow=-1
def fetchword(adr):
    """Fetches a 32-bit word from an aligned address."""
    global lastrow
    
    wordadr=adr>>2
    row=wordadr>>4;

    word=0
    for majorcolumn in range(0,32):
        # There are sixteen pairs of major columns.  If you look
        # closely as rom10.bmp, you can see that the three right most
        # bits are almost always set, because most ARM instructions
        # begin with 0xE.
        biti=31-majorcolumn
        # Like the bits being MSBit on the right,
        # the words count in from the right.
        wordi=0xF-((wordadr&0xF))  
        word=word<<1;
        word|= bits[row][16*biti+wordi];

    if row!=lastrow:
        print(";; Row %d"%row);
        lastrow=row;
    return word;

Now that OpenGL is enabled, and on some machines faster or more stable, we should make it the default.

Close this when decoding of MSP430 boot ROMs is functional.