fabricmc / mapping-io

Mapping-IO is a small and efficient library for working with deobfuscation mapping files.

Mapping-IO

Mapping-IO is a small and efficient library for working with deobfuscation mapping files. It has readers and writers for numerous formats, and provides extensive mapping manipulation facilities.

The API is inspired by ObjectWeb's ASM: At its core, Mapping-IO is visitor-based, but it also provides a tree API for in-memory storage and easier data manipulation.

Utilities for more sophisticated use cases can be found in the mapping-io-extras module; they've been moved out from the core publication due to their additional dependencies.

Usage

Reading and writing can be easily achieved via the MappingReader and MappingWriter interfaces:

MappingReader.read(inputPath, /* optional */ inputFormat,
		MappingWriter.create(outputPath, outputFormat));

The above example reads mappings from the input path directly into a MappingWriter, writing all contents to disk in the specified format. Keep in mind that the conversion process might be lossy if the two formats' feature sets differ; see the comparison table here for more details.

You can also read into a tree first:

VisitableMappingTree tree = new MemoryMappingTree();

MappingReader.read(inputPath, inputFormat, tree);

tree.accept(MappingWriter.create(outputPath, outputFormat));

If the input format is known beforehand and more direct control over specific reading parameters is desired, the formats' readers (or writers) may also be invoked directly.

Mapping manipulation is achieved either via the tree API or specialized MappingVisitors, hand-crafted or utilizing first-party ones found in the adapter package.

For further information, please consult the project's Javadocs.

Maven

Mapping-IO is available from the FabricMC Maven, version 0.4.2 and onwards can also be found on Maven Central.

Gradle snippet:

repositories {
	mavenCentral()
}

dependencies {
	api 'net.fabricmc:mapping-io:${mappingio_version}'
}

mapping-io's Issues

JOBF support

Introduced and used by JADX, with the reference implementation being available here.

An editor config is not present

Unlike other repositories on the @FabricMC organization, this repository lacks an editor config.

ClassAnalysisDescCompleter uses ASM6 so it can't read newer Minecraft jars

More specifically:

java.lang.UnsupportedOperationException: NestHost requires ASM7
	at org.objectweb.asm.ClassVisitor.visitNestHost(ClassVisitor.java:174)
	at org.objectweb.asm.ClassReader.accept(ClassReader.java:586)
	at org.objectweb.asm.ClassReader.accept(ClassReader.java:424)
	at net.fabricmc.mappingio.tree.ClassAnalysisDescCompleter.processClass(ClassAnalysisDescCompleter.java:138)
	at net.fabricmc.mappingio.tree.ClassAnalysisDescCompleter.processFile(ClassAnalysisDescCompleter.java:121)
	at net.fabricmc.mappingio.tree.ClassAnalysisDescCompleter.access$000(ClassAnalysisDescCompleter.java:45)
	at net.fabricmc.mappingio.tree.ClassAnalysisDescCompleter$DirVisitor.visitFile(ClassAnalysisDescCompleter.java:63)
	at net.fabricmc.mappingio.tree.ClassAnalysisDescCompleter$DirVisitor.visitFile(ClassAnalysisDescCompleter.java:56)
	at net.fabricmc.mappingio.tree.ClassAnalysisDescCompleter.processFile(ClassAnalysisDescCompleter.java:99)
	at net.fabricmc.mappingio.tree.ClassAnalysisDescCompleter.process(ClassAnalysisDescCompleter.java:52)

Recaf Simple support

• Reference implementation: https://github.com/Col-E/Recaf/blob/5945a28410ada589f09700843a6b4e3d60b1ae09/recaf-core/src/main/java/software/coley/recaf/services/mapping/format/SimpleMappings.java
• Enigma implementation: https://github.com/FabricMC/Enigma/tree/d93eae96a95f619d731630c6538d370ba4ae30ec/enigma/src/main/java/cuchaz/enigma/translation/mapping/serde/recaf

This is the last remaining format required for Enigma feature parity.

Flush writers in visitEnd

Improves behavior when the caller forgets to close the mapping writer.

ProGuard writer doesn't handle missing destination names correctly

Which results in outputs like this:

class_1 -> class_2 -> class2Ns0Rename:
class_3 -> class_4 -> class_5 -> class5Ns0Rename:
class_6 -> class_7 -> class_7$class_8 -> class_7$class8Ns0Rename:
class_9$class_10 -> class_11$class_12 -> class_13$class_14 -> class_13$class14Ns0Rename:
class_15$class_16 -> class_17 -> class_17$class_18$class_19 -> class_17$class_18$class19Ns0Rename:
class_20$class_21$class_22 -> class_23$class_24$class_25 -> class_26$class_27$class_28 -> class_26$class_27$class28Ns0Rename:
class_29$class_30$class_31 -> class_32 ->     int field_1 ->     int field_2 -> field2Ns0Rename
    int field_3 ->     int field_4 ->     int field_5 -> field5Ns0Rename
    int field_6 ->     int method_1() ->     int method_2() -> method2Ns0Rename
    int method_3() ->     int method_4() ->     int method_5() -> method5Ns0Rename
    int method_6() ->     int method_7() ->     int method_8() -> 

It can be fixed by passing the mappings through a MappingNsCompleter first, though ideally this would be handled automatically.

Prospect

Hello modmuss and player, I just wonder about the prospect of this project.

From a first glance, this appears to be a more efficient replacement for most available mapping libraries such as forge ones and the tiny mapping parser. Will this totally replace mapping parser, at least?

Enigma format doesn't support individual files

Even with MappingReader.read(Reader, MappingFormat, MappingVisitor) it just throws an IllegalStateException with no message.

TsrgReader crashes with an empty file

Some libraries and tools like Mixin's ObfuscationServiceFG3 produce empty files.

Example:

MemoryMappingTree tree = new MemoryMappingTree();
TsrgReader.read(new StringReader(""), tree);

// Leads to:

Exception in thread "main" java.io.IOException: missing class-name-a in line 1
	at net.fabricmc.mappingio.format.TsrgReader.read(TsrgReader.java:100)
	at net.fabricmc.mappingio.format.TsrgReader.read(TsrgReader.java:56)
	at net.fabricmc.mappingio.format.TsrgReader.read(TsrgReader.java:52)

Implement a tiny v1 writer

Sounds quite legacy, but when trying to migrate yarn tasks to mapping-io this becomes a roadblock.

TsrgFileReader doesn't support flat structure

Format example:

test/TestClass myField a
test/TestClass myMethod ()V b

Mixin's MappingWriterTSrg writes these files so it'd be nice to be able to read them as well.

Enigma reader silently ignores invalid lines

Example: (indented with spaces instead of tabs!)

CLASS a Foo
    CLASS b Bar

If the file above is visited with a MemoryMappingTree, the output will just have an entry for a and not say a word about the invalid data for a$b.

Allow multiple source namespaces

related to fabric-loom#1026

To my understanding, the current Tiny format allows for a single source namespace, which for classes consists of the source name, and for fields and methods the source name and the source descriptor. Then there can be any number of target namespaces, which for all types is just the target name.

To allow merged mappings for pre-1.3 Minecraft versions, however, two source namespaces are required, for the client and server sides. The client and server are obfuscated differently for these versions, so the names and descriptors do not match, and both will need to be listed.

If MIO alllowed for this, then Loom could properly support merging the client and server for these versions.

IntelliJ Idea Migration Map support

Just a basic XML file, with examples being available here, here and here

Allow setting src name for mapping tree elements

It's weird that people can set the dst name but not the src ones.

provide an outer class fixing visitor

this visitor reads in two passes, first, it collects class mappings. then, for each null dst name of a nested/inner class, it will replace their name with auto-filling the mappings of outer class names from a given namespace. should be useful as a few places in yarn and loom both need this.

XSRG and CSRG support

• XSRG spec: https://github.com/MinecraftForge/SrgUtils/blob/67f30647ece29f18256ca89a23cda6216d6bd21e/src/main/java/net/minecraftforge/srgutils/InternalUtils.java#L69-L84
• CSRG spec: https://github.com/MinecraftForge/SrgUtils/blob/67f30647ece29f18256ca89a23cda6216d6bd21e/src/main/java/net/minecraftforge/srgutils/InternalUtils.java#L196-L207 (basically TSRG, but flat, see #39)

Apart from the linked SrgUtil implementation, there's also the Lorenz one for reference and with test files.

Sorting mappings

I am trying to migrate yarn to use mapping io than stitch. In the process, I find that mapping io's output order is arbitrary; since it's quite hard to sort mappings manually, I wish mapping-io can allow memory mapping trees to be sorted by, say, src names, and have easily debuggable outputs (the previous outputs of yarn are alphabetically ordered as well, and @sfPlayer1 has proposed to use the order as a way to quickly peek classes for a file-backed mapping tree)

Make tiny-remapper an optional dep

Right now MIO depends on tiny remapper, as TR is not on maven central you must also use the fabric maven to resolve TR. We should remove TR as a dep from the maven pom.

MemoryMappingTree exposes private types as return types

...which makes the methods annoying to use since you have to store them in a local that forces them to the public API type.

Examples: getClasses() (generic in Collection<ClassEntry>), getClass, addClass, removeClass etc and similar methods in the entry implementations (MemberEntry.getOwner, ClassEntry.getMethods/getFields etc).

JAM support

• Former name: SRGX (not to be confused with XSRG)
• Specification: https://github.com/caseif/JAM

Pretty obscure and apparently not used anymore, but Lorenz has support for it, so providing at least a reader for this format would be nice.

Add support for return-less method desc

Java doesn't allow methods with the same name+parameters but different return types in the same class. This should be supported where appropriate, potentially special casing method descs ending with ), similar to the desc-less field support.

Merging Enigma files into trees deletes dst names

Causes FabricMC/fabric-loom#714.

Example:

mappingsA/Test.mapping

CLASS a Test
	CLASS b Foo
		FIELD a foo Ljava/lang/String;

mappingsB/a.mapping

CLASS a
	CLASS b
		FIELD a bar Ljava/lang/String;

Read mappingsA, then mappingsB into a MemoryMappingTree.

Expected: a$b has the dst name Test$Foo
Found: a$b's dst name has been erased by the second read call and replaced by a$b (or Test$b? I might've made a typo here)

Calling MappingReader.read with an unsupported format results in an undescriptive exception

When one of the read methods is passed an unsupported format (apart from MCP), the code throws an IllegalStateException with no message:

mapping-io/src/main/java/net/fabricmc/mappingio/MappingReader.java

Lines 190 to 191 in 597f072

	default:
	throw new IllegalStateException();

IllegalStateException isn't really appropriate here ("Signals that a method has been invoked at an illegal or inappropriate time."), and the user gets no useful information from the exception since it has no message either.

'queuePendingMember' causes NPE

There is this bit of code here, which constructs FieldEntry and MethodEntry objects with the source name set to null:

mapping-io/src/main/java/net/fabricmc/mappingio/tree/MemoryMappingTree.java

Lines 479 to 483 in a301899

	if (isField) {
	member = new FieldEntry(currentClass, null, desc);
	} else {
	member = new MethodEntry(currentClass, null, desc);
	}

I think this always causes a null pointer exception, because the parent constructor uses it to construct a key:

mapping-io/src/main/java/net/fabricmc/mappingio/tree/MemoryMappingTree.java

Line 1122 in a301899

this.key = new MemberKey(srcName, srcDesc);

And the constructor of the key in turn calls .hashCode() on the name without checking for null:

mapping-io/src/main/java/net/fabricmc/mappingio/tree/MemoryMappingTree.java

Lines 1680 to 1684 in a301899

	if (desc == null) {
	hash = name.hashCode();
	} else {
	hash = name.hashCode() * 257 + desc.hashCode();
	}

I suppose, since a key without the name being present probably makes little sense, it would make sense to lazily compute the key as soon as the entry is no longer pending?

Stack trace of the exception:

Caused by: java.lang.NullPointerException: Cannot invoke "String.hashCode()" because "name" is null
	at net.fabricmc.mappingio.tree.MemoryMappingTree$MemberKey.<init>(MemoryMappingTree.java:1683)
	at net.fabricmc.mappingio.tree.MemoryMappingTree$MemberEntry.<init>(MemoryMappingTree.java:1122)
	at net.fabricmc.mappingio.tree.MemoryMappingTree$MethodEntry.<init>(MemoryMappingTree.java:1217)
	at net.fabricmc.mappingio.tree.MemoryMappingTree.queuePendingMember(MemoryMappingTree.java:482)
	at net.fabricmc.mappingio.tree.MemoryMappingTree.visitMethod(MemoryMappingTree.java:459)
	at net.fabricmc.mappingio.adapter.ForwardingMappingVisitor.visitMethod(ForwardingMappingVisitor.java:77)
	at net.fabricmc.mappingio.format.tiny.Tiny2FileReader.readClass(Tiny2FileReader.java:149)
	at net.fabricmc.mappingio.format.tiny.Tiny2FileReader.read(Tiny2FileReader.java:116)
	at net.fabricmc.mappingio.format.tiny.Tiny2FileReader.read(Tiny2FileReader.java:55)
	at net.fabricmc.mappingio.MappingReader.read(MappingReader.java:186)