thpatch / thcrap Goto Github PK

Improved font replacements

Description

Currently, we merely have a single replacement font which overrides any font face names in CreateFont. This usually is enough for Touhou.

However, selecting a replacement font based on various parameters of the original font (face name, size, width, etc.) would allow for more detailed configuration. Text originally rendered in a smaller font could be made larger without affecting regular text, and fonts which previously looked too small would become usable.

An example of how this may look like:

{
    "fonts": {
        "'ＭＳ ゴシック'": "'UVN Chim Bien'",
        "'ＭＳ ゴシック' 10" : "'UVN Chim Bien' 16"
    }
}

Prerequisites

None.

Implementation

Multi-slot breakpoints

Description

This will allow one breakpoint function to be triggered at multiple positions, without having to register an arbitrary number of breakpoints for the same function.

Prerequisites

None.

Implementation

• Deprecate manual breakpoint registration via breakpoint_register. Breakpoint functions now must be exported and have a name of the form BP_<name of breakpoint>. Hooray for enforced coding conventions.
• In the breakpoint name, make # work as a sentinel character, splitting the breakpoint name and the slot name

CRC32 verification

Description

This seems to be the best way to gracefully handle broken or incomplete mirrors, and could also be the foundation of a security system (#10).

Prerequisites

None.

Implementation

Has been requested multiple times in multiple places. Will have to look into this once we have the time.

Endings

Description

For time reasons, we will probably just focus on the custom MSG opcodes for endings used from th10 on.

Prerequisites

• #17 (text layout engine)

Implementation

This would only download the most necessary data of each patch (game version data in particular) on initial patch selection, with the rest being only downloaded when needed (e.g. once a particular game has been located on a user's system).

In particular, this means:

1. Only pull versions.js for every patch after stack configuration
2. Use that to start searching for games
3. Download any global patch data during the search process
4. Create shortcuts
5. Download/update the rest of each game's data individually after the user started a game through a shortcut, using a blocking downloader.

Patch dependencies

Description

More connection between the individual patches.

For example, translation patches into Arabic or Hebrew could depend on a shared patch adding support for right-to-left text. In this case, the user would not have to bother downloading both base_tsa and, let's say, script_rtl, but these would be immediately added and stacked below the translation patch, if they aren't already there.

Also, we could split off the font settings and font files currently bundled in base_tsa into an own patch (script_latin).

Prerequisites

We'll probably wait for #8 (GUI configuration tool) before attempting something like this.

Implementation

Currently we cannot use vsyncpatch(vpatch) or other existing patches with thcrap because they also require DLL injection.
Possible implementations may be CreateProcess hook or other similar things.

ZIP archive support

Description

This will be necessary for PC-98 support, if only to work around DOS 8.3 filenames.

Prerequisites

None.

Implementation

Advanced image patching

Description

The basic image patching implemented in #12 doesn't have dedicated support for patch stacking. This feature goes a bit further and adds support for overlays and sprite-level replacements to save both redundancy and needless bloat.

Prerequisites

• #12 (basic image translation)

Implementation

See the development notes on Touhou Patch Center.

I've been wanting to do this for a long time, and half of fast bootstrapping would require it as well. This is how it could work:

• When thcrap_loader.exe is started, it checks whether thcrap_downloader.exe exists and launches it if it does. Otherwise, it goes straight to launching the patched game, as it does currently. This will continue to allow people to opt out of updates.
• thcrap_downloader then determines whether there are any updates and how much to download in a blocking manner before starting the game itself.
• Once the game is started, the downloader process remains open in the background to fetch new updates while the game is runnning.
• It also remains open after the game was closed, in order to inform users about the updates that happened in the background.

The updater needs to block in the following cases:

• Technical patch changes that can't easily be repatched while the game is running, such as binary hacks and breakpoints.
• Any changes to the dependencies of selected patches, as this might potentially require user interaction (see below).
• The initial bootstrapping process of a patch. (It may be better, though, to instead define some sort of threshold value above which a patch update becomes generally blocking. This would also help to accommodate scenarios where users quit the initial blocking download after a few files, then later restart and wonder/complain why next to nothing is patched because the downloader is still running in the background...)

Reasons for putting the downloader into a separate process, rather than just keeping it in a thread within the game process:

• Stability. In rare cases, the download thread seems to cause a crash before the title screen, and it also tends to crash if the user immediately quits on the resolution dialog while an update is in progress.
• Antivirus software would be less likely to target thcrap. I looked into the situation with Norton, and it only killed the game process on the HTTP GET requests made by the uploader.
• Users would not have to add firewall exceptions for every single game executable, but only once for the downloader.

Reasons why this absolutely must have a GUI:

• Progress bars are obviously prettier.

• The Windows console subsystem still can't render Unicode properly – or at least not without requiring extensive configuration together with correct fonts, which I personally have never got working.

• We could add a nice display of the differences between the newly downloaded content and the existing files, so that users can easily see what changed in the current update. (I can already imagine that people would then be calling for some sort of spoiler protection, though?)

• We could add a "Start game now" button that is enabled once the downloader has finished all blocking parts of an update, as an alternative to just immediately running the game in that case.

• User interaction will be necessary to manage digital signatures on patches, once we have them.

• We could also give users a choice between available alternative patches that offer the same features. This would implement our new focus on namespacing, and is especially important for changes to the underlying dependencies after configuration time. (Really, it's about time that our design acknowledged the fact that maintainers of anything Touhou are most likely not going to stay around forever.)

  As you know, dependencies are stated using the syntax

  [repository/]id
  

  It is already possible to omit the repository part. Currently, this results in the patch being randomly picked from a repository that offers a patch with the same ID. With user interaction, we could instead ask the user which patch to choose.

  This question should come with a detailed overview of both patches, (rendering each patch's readme.md next to each other would do). This allows users to make a somewhat informed decision between two patches implementing the same functionality, even if they haven't followed the development of the patches in question (which they most likely haven't).

Things to consider

• Looking at Steam, it seems that gamers have largely accepted updates delaying game startups from time to time. If that's the case, we could just block on every update to simplify the design and the implementation.
• How much overlap will there be with the future configuration tool? If the answer is "a lot", why don't merge them outright?

th06 ending format

Description

Prerequisites

None.

Implementation

For reference, I'm running 64-bit Windows 8.1. I have DDC installed at D:\Touhou\Touhou 14 - Double Dealing Character, and thcrap is at D:\Touhou\thcrap\ .

I bought and downloaded the full version of DDC from Playism, and I downloaded the latest version of thcrap from thpatch.net and ran it. I selected the English language patch and pointed it to the right directory, and it generated shortcuts that seem correct. I can run the shortcut to custom.exe without any problems, but when I try to run th14, it crashes immediately. I can run th14 by itself without any problems.

Description

Quite a number of people have commented that integrating the thcrap shortcuts into Steam is a bit unintuitive. This point has become particularly relevant now that ZUN is investigating official download releases. In the future, Touhou games might very well be distributed through Steam, making a proper integration all the more desirable.

Prerequisites

None.

Implementation

You would think that Steam offers an API for managing the shortcuts to non-Steam applications. But no, you have to parse the Windows registry and manually write binary files to edit the shortcut list. The Ice project seems to be a "reference implementation" of how to achieve something like this - we'd only have to adapt its code.

Right-to-left text

Description

It was only a matter of time until people asked for this.

Prerequisites

• #17 (text layout engine)
• #16 (Win32 dialog patching)

Implementation

• If possible, this should not require a special "rtl": true flag in the run configuration. Directionality should be automatically determined by the characters in a string. Otherwise, patch stacks that have a RTL language on top of an LTR one couldn't easily switch between both.
• Neither should it require all texts to simply be wrapped in a <r$text$> command. Once tabs come into play, we need to mirror those too.
• Add a breakpoint to read the current width of the rendering bitmap.
• According to this page, Win32 dialogs can be easily mirrored. Find the best way to do this.
• ... and then there's texture positioning. We probably could start by searching for a general "set X position" point and mirroring that value. Anything more will be part of a separate issue, though.

Dynamic base addresses

Description

Right now, thcrap_loader, the binary hack system and the breakpoint system all assume a game's main module to be loaded at a constant place in memory. Thus, a game loaded to a dynamic base address essentially can't be patched at all.

This feature is required for supporting Hopeless Masquerade and the PC-98 games at all.

Prerequisites

None.

Implementation

• For binary hack and breakpoint addresses, add rva as an alternative key for addresses relative to the module's base address.

Hardcoded strings

Description

"Hardcoded strings" here means "immutable strings stored at a constant address in memory". This allows us to do a lookup solely based on the address.

Prerequisites

None. Also needs to work with #3, though.

Implementation

This consists of four parts:

1. Basic lookup function based on a string's address
2. Hooks for TextOut, MessageBox and possibly other Win32 API functions calling the lookup function
3. Breakpoint to call the lookup function (unnecessary; hardcoded strings are either fully constant (which means our hooks take care of them anyway) or format strings (see below).)
4. Custom sprintf handler

No. 4 is necessary because some of these hardcoded strings are sprintf format strings, and we shouldn't trust on the game's own buffers being large enough.

US and Japanese do this: 9,999.99
Some European countries do this: 9.999,99
We need to support both ways.

We need one. I choose NSIS.

Code signing

Description

Marking as both a bug and an enhancement because it may become really scary one day.

First of all, I must admit that I am a total newbie to this concept, and I'd like to thoroughly understand anything (and, most importantly, why it works or doesn't work) before implementing it.

This is disquieting not mainly because the patcher executables may be tampered with, but because every patch can contain self-updating binary hacks.

Everyone with write access to the JSON files of a patch can have code execution in the context of the game process. While this openness is certainly one of the main aspects about this project, it can just as easily be abused.

The main problem here is that we want to keep a low barrier of entry for this whole patching thing on the technical side too (even though this is hardly used), yet also want to ensure security somehow.

Once we have a working system in place, we can also have patches that come with self-updating plug-in DLLs.

Prerequisites

• Probably requires Flagged Revisions on the wiki side, combined with a way to automatically sign a patch file upon flagging, to really work.
• If automatic signing based on wiki logins even is a good idea in the first place.

Implementation

• Provide two separate versions of thcrap on thpatch.net: A regular version with all the code signing enabled, and a developer version that doesn't have these features. The latter will have a big "this version may pose a security risk, only use it if you know what you're doing warning.

With the reorganization of the project and the progress we've made in the last year, I think it's worth raising this question again. The new project administration will need to take a stance on this before covering the next game, and it'll be important to know the background behind my decision against such a system.

It has always been my design goal for thcrap's translation functionality not to rely on any original Japanese strings. This decision stems from two observations:

• There are static translation patches for earlier games into a variety of languages.
• With the older games essentially being abandonware [citation needed] and, as of this writing, only one game being distributed as an official download release, piracy remains the only distribution channel that really matters. And it has been shown time and time again that most site owners rather provide complete, easy-to-use packages aimed to speakers of their own language than clean, original copies - which is further justified by some inadvertent "region locking" that e.g. causes 東方紅魔郷.exe and every game's custom.exe to simply not work on non-Japanese locales.

Combine the two and you have rampant piracy of unofficially translated games. We can't rely on pirated copies coming with the original text anymore. That's just how it is.

This means, however, that we have needed all sorts of alternative indexing systems to correctly assign translations:

• Dialogue and endings use a rather involved time code system that additionally gives translators the freedom to edit dialogue on a text box level and allows them to use 2 lines where the original only uses one (and vice versa). This greatly goes against the line-centric design of ZUN's original formats, and is one source of all the complexity in the dialogue patcher (with the other one being the hard line system used in th06, th07 and parts of th08).

• Spell cards use their (zero-based) index number in the game's Result screen, which are pulled out from the game's memory using a bunch of breakpoints. This works fine for games that have a result screen. All games that don't (TH095, Uwabami Breakers, TH125, TH143) coincidentally also happen to have no reliable index numbers, and therefore require even more build-specific binary hacks and breakpoints to somehow derive them from the game progression - or, in the case of Uwabami Breakers, a full copy of all .ECL danmaku scripts with corrected IDs. :(

• Hardcoded string translation assigns IDs to virtual memory addresses (stringlocs.js), then looks up translations for these IDs in a separate table (stringdefs.js). This is reliable, needs zero game- and build-specific hacks, and I have a script to locate the addresses, but they still need to be committed for every build of every game.

• Music Room translation uses a combination of hardcoded string translation (for the "No. X ??????" strings displayed for locked tracks), a separate file containing song title translations (themes.js) and a separate file for comments (musiccmt.js).

  The themes.js system was designed before thcrap to serve as a song title source for (hypothetical) third-party applications to cope with frequent translation changes (which in turn was the main motivation for thcrap in the first place).

  musiccmt.js basically uses the same format as the generic plaintext translation support that would later be developed for th143, but with a special syntax that replaces a single @ character in a line with a customizable format string, printing the title of the currently selected track.

  Not to mention that pulling the theme number out of the game also requires its fair share of build-specific breakpoints.

• Dialog resource translation makes use of the fact that the widgets (and thus, their strings) internally appear in a set order, builds a JSON array of hardcoded string IDs (dialog_*.js) in this order, then pulls the translations out of stringdefs.js. Other than that, no build-specific hacks necessary.

Using one single dictionary-based solution instead of these systems would greatly reduce the amount of effort required to support new games at the expense of both compatibility to static patches and more bloat in the translation files.

Text layout engine

Description

This is at least necessary for horizontal alignment of text in the endings and the Music Room. th11 would also benefit for the assist character dialogue.

Prerequisites

None.

Implementation

Reverse-engineer the layout format used by the th13 English patch and add MediaWiki's syntax for bold and italic text - maybe even for Ruby.

Listing this as wontfix. Time spent trying to do this could be better spent elsewhere. If ReC98 is not released and has had no development for 2 straight years we will reconsider supporting the PC-98.

Requires OllyDbg64 and #39 with MinGW support.

Relative paths

Description

Prerequisites

None.

Implementation

Dynamic linking of thcrap_update in the configuration tool

Description

We want users to opt out of any automatic updating functionality of both the patches and thcrap itself by simply deleting thcrap_update.dll. This is no problem as far as the base engine is concerned, but the configuration tool has a static link to this DLL.

Prerequisites

None.

Implementation

Resolution dialog

Description

A fairly low priority, not even included in the C85 milestone.

Prerequisites

Implementation

This would fix and improve a couple of things.

Hot-repatching

Description

With this feature, newly downloaded binary hacks, breakpoints and other patch data loaded on initialization could immediately be re-applied after a patch update was downloaded, without having to restart the game.

Prerequisites

• Breakpoint removal
• File change notifications also need to work with ZIP files, preferably without having to care whether patch files comes from a ZIP or a real directory structure.

Implementation

• Each plug-in (and the core) can export a thcrap_plugin_repatch() function.
• thcrap_repatch() calls each DLL's thcrap_plugin_repatch().
• Using the file change notification functions of the Win32 API, this function is also called whenever the user changes a file locally.

Currently, thcrap_configure resolves patch dependencies and saves a complete stack in the generated run configuration file. By instead doing this every time a user starts a game, we'd only have to store what the user actually selected. This also allows patch maintainers to freely split or fork one patch into multiple ones, without requiring every user to reconfigure.

Prerequisites

Needs a blocking updater that runs in a separate process (#37). We can't start running the game before we know the exact patches the user needs to use.

Display the contributors of patches in a more accessible way.

This is for thcrap itself, not the patch updater.
The updater should be able to delete obsolete files.

Images

Description

Prerequisites

Implementation

This would allow text replacements on a finer granularity than the current translation units, and could e.g. be used to simplify the implementation of patches like the British English translation.

This issue was closed on Trello with reference to the dictionary system in #38. However, that system would only operate on complete strings pulled from the game, which match the current translation units. Sure, the regex features will be made available to translators through a dictionary system (perhaps even in the same file), but the regex part still has to be implemented separately from that.

Feel free to close/wontfix this issue now if this is still something you don't want to have, just wanted to clarify that.

Spell cards

Description

Basically, "everything translatable in an ECL file" - also includes the spell card comments in th08, but not in th095 or th125.

Prerequisites

• Issue #1 (multi-slot breakpoints)

Implementation

ID-based lookup table, accessed via breakpoints. No reason to start parsing and patching ECLs here, and also nicely works around can include a different fix for the "in the result screen, spell cards appear in the language they were last encountered in" issue.

Translated names will be stored in game_dir/spells.js, comments in game_dir/spellcomments.js.

Instead of directing your users to the thcrap download site, you would simply have a custom URL like

thcrap:bootstrap:https://raw.githubusercontent.com/Gamer251/thcrap-gamer251/master/mima/

and clicking that would automatically download a patch and create shortcuts.

This has the advantage of true decentralization: We would no longer need the approval of another node in the network that lists a certain repository. Instead, people can download the patch directly from your site.

The basic code to register such an URI scheme is very simple. You just have to add a bunch of registry keys and make thcrap respond to those parameters. This can be done automatically each time you run thcrap.

This might seem to forgo patch stack configuration entirely in favor of usability, but I can think of two ways to fix that:

One way would simply run thcrap_configure normally, but ensure the given repository to be present in the patch list:

thcrap_configure:https://raw.githubusercontent.com/Gamer251/thcrap-gamer251/

In fact, since all required functionality for this is already present in thcrap_configure, I could add this scheme immediately.

Another way would be to provide multiple pre-configured stacks on your site, which all would be able to be set up with one click:

thcrap:bootstrap:thpatch/lang_en https://raw.githubusercontent.com/Gamer251/thcrap-gamer251/master/mima/

This would automatically add the English patch below the Mima patch.

As you see, you can either give the complete URL to a patch, or its repo/patch ID.

With the repo/patch ID combination, thcrap would of course need a way to actually locate the patch download. For "well-known" repositories like thpatch or nmlgc, this is no problem - these are always included in the distribution. Otherwise, thcrap will do a recursive neighbor search across all the servers it knows - and, what makes this really powerful, also all the servers directly specified as URLs in the link.

This means that repositories simply need to list URLs of all the servers they need as its neighbors (which is best practice anyway), and you're guaranteed to get the right patches.

Step 1 for supporting alternative compilers.

Right now, this is undefined behavior, but who knows, maybe this might come in handy some day.

• Don't bundle all the games in one patch. Split them up per game, per translation/function.
• Don't download each file separately. Bundle them up in a zip.
• Namespacing.
• While we are at it, change .js and .jsdiff extensions to .json.
  • For jsdiff, either document the existing markup better or change it to something like markdown.
• There is probably something I forgot about, I'll add it later if so.

When we get a GUI and we have thpatch 2.0 non-mediawiki edition, display the progress bars based on the information served from thpatch.net

Allow patches to contain DLLs which are loaded into the game process. This will drastically enhance what patches can do: For example, the British English patch could be generated automatically by adding a DLL that applies a set of text replacement rules to every line of text delivered from the engine.
#10 is a prerequisite for this. We should not add a feature that executes arbitrary binaries downloaded from arbitrary servers without a way to prove their authenticity.

JSON reload option

Description

Prerequisites

None.

Implementation

Music Room

Description

Prerequisites

• #17 (text layout engine)

Implementation

i18n

Description

Isn't it ironic that we can translate games, yet can't translate ourselves?

Prerequisites

• A better way of templating and outputting strings than good old printf. I'm open for any kind of suggestion here - even changing the project's programming language, if you can convince me.

Implementation

Window title translation

Description

Sounds trivial at first - just use string table lookup and you're done, right? However, if you want to...

• minimize redundancy between game versions (we don't want to change the full string for every version)
• minimize redundancy between languages (most will probably choose to romanize the Kanji part while keeping the English part in English)
• integrate that into the wiki while still only using the title parameter in the run configuration
• and also print some additional information (e.g. the thcrap build and the currently active patch stack)

..., this becomes quite involved.

Prerequisites

• #18 (Patch dependencies)

Implementation

th08 spell owner

Description

Just so I don't forget, since #15 is done now. :)

Prerequisites

None.

Implementation

Ruby layout

Description

By re-implementing ruby text display using the text layout engine from #17, we could get rid of the guessing code in the wiki extension and the need for a binary hack for each game to disable ZUN's 2-byte fixed-width ruby loop.

Prerequisites

#17, obviously.

Implementation

GUI

Description

This requires careful planning: an interface that allows people to do everything the engine is capable of would result in something even harder to use without careful design.

The GUI needs to support:

• Multiple originating servers
• Blacklisting of files per patch
• Font settings
• A simple-to-use, verbose and clear wizard
• A way to suggest base_tsa, yet also allow for configurations that don't use it Dependencies
• Enable and disable self-updating
• more... ?

Prerequisites

• Settle on a GUI framework (decided on wxWidgets)

Implementation

Patches are visualized as horizontal blocks and colored according to their downloading status. These blocks contain little icons symbolizing the various types of data included in a patch:

• Version information
• Binary hacks and breakpoints (may share one icon, maybe?)
• Fonts
• Text translations
• Images

Maybe we could also indicate progress in some way?