col-e / recaf-3x-issues

This was the temporary issue tracking for 3x. Focus is now on 4x and all issues are tracked on the main repo going forward.

Recaf 3X

▶️ Supersceded by 4x ◀️

Recaf 4X combines a backend rewrite and the new features introduced in 3X and puts them together.

Work going forward will only be on 4X. Effort is being put into getting it up-to-par with where 3X left off ASAP.

This repository is a temporary one for staging snapshot previews of the 3X branch. It also helps organize issues by delegating them to one place without cluttering up issues affecting the current 2X ones in the main repo.

The purpose of these snapshot previews is to gather feedback and opinions. Try it out and let us know what you think about it.

Download

Get it here: 3X Snapshot releases

Important links

• Source: 3X branch
• Issues: 3X issues
• Discord:

FAQ

• Is 3X usable right now?
  • Yes, but we're going through testing of the main features in order to catch bugs prior to an official release
• When will 3X be done?
  • Projects like this are never "done". They just improve with time. That being said, 3.0.0 will be released when all of the basic features are implemented and no obvious bugs appear during regular usage.
  • The major remaining components needing bug-testing and validation are: Instrumentation (Agent/attach), SSVM
  • Documentation for 3X needs to be created over on Recaf Documentation
• What will the 2X --> 3X update process look like?
  • We will create a final 2X release that will change the update process to do the following:
    • Notify that the next update to 3X will bring significant changes (But in a good way)
    • Notify that Java 11 is required (Skipped if user is on Java 11 already)
      • Warn about JAVA_HOME in addition to current JVM if they are not the same path.
    • Prompt to update to 3X, or to ignore the update all together.
    • Upon acceptance, setup usage of the Recaf 3X launcher instead of Recaf 3X directly so that the update process is handled externally
    • Close 2X, run the new launcher.
  • In case the user uses an old version, its really on them if the update process doesn't work.
    • It will attempt to do java -jar <recaf> on the new jar, which should work if they're already on Java 11. Though they will not be using the launcher in this case.
    • If they are on Java 8, it will probably fail in some regard and be up to the user to resolve.

Progress

This table is a very brief overview of the current state of 3X. It does not cover all of Recaf's existing features, nor does it cover all of the upcoming new features.

Feature Name	Completed?	Progress
Assembler	✔️	Feature complete, focusing on fixes.
Android support	❔	Basic Android support exists for APK files, but a lot of the UI has not been fully integrated, so there's some menus that do not function.
Compiler support	✔️	Classes in the workspace can be modified via decompiling and recompiling.
Attaching / Agent support	❔	The basic capability is added, but needs some polish to fix up bugs and clear up UX.
Decompile support	✔️	Decompiler integration is in. Currently supports CFR, Procyon, QuiltFlower, and fallback options. Contextual actions are available on classes, fields, and methods assuming the decompiled code is parsable.
Hex editor support	✔️	The hex editor has been redesigned from the ground up. It now comes with a string dumper, value inspector, and other helpful tools for class files.
Natives / PE/ELF viewer	✔️	There is support for viewing both PE headers (DLL/EXE, most useful for imported functions) and ELF headers.
Inheritance graphing	✔️	When viewing a class you can see its parent and child classes/interfaces
Manual remapping support	✔️	Remapping is internally supported and integrated. Right clicking on classes/fields/methods allows renaming of each item.
Automated remapping support	✔️	Batch remapping from mapping files is implemented, though file formats will need to be ported over to the new system.
Generative mapping suppor t	✔️	New mappings can be generated on the fly with the new built in auto-mapper. You can specify filters to narrow down what gets mapped and the kind of naming scheme you wish to apply.
Persistent config	✔️	The system for saving configuration values is done. Some UI elements are missing though.
Dockable panels	✔️	In 3X all tabs have built in docking support. You can drag a tab out of the window to create a new window, snap tabs to split a window into halves, and more.
Phantom class generation	✔️	The feature that lets you generate missing classes (Used heavily to enable compiling against code you don't have) is fully ported over.
Custom themes	❌	No progress has been put into supporting changing the default dark theme yet.
Custom translations	✔️	Multiple languages are almost fully translated and swapping between them has been implemented.
Virtualization	✔️	Virtualization driven method call simulation and code deobfuscation are available in the UI. SSVM powers the virtualization. Since native API's are re-implemented in Java only supported systems can be virtualized (Which is good from a security standpoint).
Plugin support	✔️	The plugin API has been fleshed out, and the plugin manager is accessible via the UI. More things like browsing public plugins, a more fleshed out API, etc will after the first release of 3X.
Script support	✔️	The script menu lets you edit/load files and execute them using Beanshell. -s <file> command line flag runs a script on startup.

recaf-3x-issues's Issues

Port null terminator patch in decompile view from 2x to 3x

Col-E/Recaf#409

Update Drone YML to use Node 14 instead of 13

From recent runs:

1	13: Pulling from library/node
2	Digest: sha256:70d4fffcab39a1f9f7161d58e674ddcc56c7f0724196b68d52a87bab15cb4a04
3	Status: Downloaded newer image for node:13
4	+ npx semantic-release
5	npx: installed 577 in 28.627s
6	[semantic-release]: node version >=14.17 is required. Found v13.14.0.
7	
8	See https://github.com/semantic-release/semantic-release/blob/master/docs/support/node-version.md for more details and solutions.

Drag new tab into new window, close window, and you can no longer open new tabs

The DockingRootPane's Stack<DetachableTabPane> recentTabPanes is not being updated when a tab-pane is removed.

Improve paragraph factory system to allow re-use of paragraph graphic entries for different purposes

Right now the paragraph factory generates line numbers and then problem indicators.

Change this so there are two parts:

• Line numbers
• Shape suffix

Here's draft picture of the goal

Problem indicators still will show as red triangles. But then we could also add variable reference indicators and other things. The triangle shape is re-used. So the problem indicator and variable indicators will both have a reference to the triangle which is normally invisible, and are able to change the color based on user interaction.

Some potential cases:

• Error indicators
• Lines where variable is also referenced (also being shown in a tab with a table would be useful)
• Jump taken destination
• Jump not taken destination
• Variable scope (Not using triangles, could be a thin line to the left of them, or changing the line number bg color) of current selected variable
• Search result matched lines

Call cleanup() on closed tabs

There is already code for this in DockingTabPaneFactory, but the docking framework cannot tell the difference between closing a tab vs dragging it out of the current pane and into another.

Some ideas:

• Find an alternate way to detect tab closure
• Open a PR in the docking framework to support "Intent" (Where close could be via X, programatically, or something like dragging into a new panel)
  • Only act on the actual closing intent, not things like drag/drop

Ability to remove recent items from the recent files menu

Adding an X button would be nice to clear up clutter in the menu once you're done looking at a file.

Allow stripping debug before passing to decompilers

Decompilers may emit ugly code due to junk in the debug information (Like mineccraft's snowmen characters).

There should be a way that lets users strip debug from the classes so they decompile cleanly. Some ideas:

• Rework decompiler interfaces to incorporate these options (setStripDebug(bool))
• Let users strip debug information from any class/package

Updates in the workspace should automatically update any related UI components

The GUI presentation layer's workspace sub-layer should have all the listeners we need for this. When a class is updated, any open tab representing the class should be updated. Same goes for generic files.

Classes/Files that are removed should also be closed.

Better image handling when selecting them from the file tree

In 2x the image is displayed in the center and is not interactive. Having some basic features would be useful:

• Zoom in/out with mouse-wheel
• Pan with click and drag

Having some sort of in-between animation between decompilers to reduce "jitter" would look nice

When a re-decompile happens the text is cleared, then set again which fucks up the scrollbar.

There is a CodeArea.replaceText(String) but it doesn't play nice with the CSS restyling (for now)

• Plus, maybe we don't have to replace the whole document if most of it matches. Can do keep the first chunk of the document that matches and the last chunk. Leaving only the middle region to be replaced and restyled.

Search panel / results

Create the search panel and add it to the relevant context menus (different panel setups for different searches)

When this does get implemented it should open in a new window as a dockable tab. This way users can drag the tab into some other region of Recaf if they want to keep a search window open at all times, rather than have a separate floating window.

Constant pool editor side panel for Java classes

Next to field/methods & inheritance add a tab for constant pool editing. This will provide quick access to lightweight changes using CafeDude. Due to the simplicity of editing one item at a time it should also be rather resilient to obfuscation.

Pull out "center flowless cell on screen" logic to util

Used in HexView and SyntaxArea.

Allow assembler to modify class-level attributes

Could add a new rule for classes in the bytecode.g4 which is could use DEFINE-CLASS instead of just DEFINE so the token is different.

From there it can support modifiers and annotations like a normal field/method assembler rule-set

Ability to insert labels and automatically adjust adjacent ones

A:
LINE A 10
  // foo
B:  <------ Right click here?
LINE B 11
  // bar
C: 
LINE C 12
  // baz

• Insert line (55)

A:
LINE A 10
  // foo
B: 
LINE B 55
  /////// inserted
C:
LINE C 11
  // bar
D: 
LINE D 12
  // baz

Without feature would need to manually add new label, which does not follow ABC pattern:

A:
LINE A 10
  // foo
INS: 
LINE INS 55
  /////// inserted
B:
LINE B 11
  // bar
C: 
LINE C 12
  // baz

Without "System.exit()" closing the main window wont end Recaf if a class has been viewed

While slapping in System.exit(0) resolves the problem, I believe there is something not being cleaned up that should be addressed.

There shouldn't be any extra threads running in the background or anything... 🤔

Support node resolution of import statments

Add an edge case for JavaParserResolving for import statements.

• We get a Name node but the parent is ImportDeclaration
• import static com.example.Foo.*; can't be resolved
• but import com.example.Foo; can be and so can import static com.example.Foo.myStaticMethod; in some cases (As long as its not an overloaded method)

Support ZIP entry [Comment] and [Extra] metadata

Recaf does not record this information when reading from ZIP archives. So when a file is exported this data is lost.

This breaks some unique obfuscations, such as the one employed by ItzSomebody in the 2021 RedPwn CTF

• Bug reported - https://ctftime.org/writeup/29140
• Relevant sample - https://github.com/redpwn/redpwnctf-2021-challenges/blob/master/rev/javaisez3/bin/javaisez3.jar

What happened with the class mode "Table"

It was really useful since decompiler gives me a lot of errors.

Handle frame-generation with Java 7- classes with JSR/RET

Duplicate of Col-E/Recaf#357 to make sure it gets resolved

Update RichTextFX to fix spam of "Visible paragraphs' last index is [-1]"

The bug is present in 10.6 but was fixed 2 months ago. When a new release is made it should fix the spam: FXMisc/RichTextFX#1021

Full log

java.lang.IllegalArgumentException: Visible paragraphs' last index is [-1] but visibleParIndex was [0]
	at org.fxmisc.richtext.GenericStyledArea.visibleParToAllParIndex(GenericStyledArea.java:982)
	at org.fxmisc.richtext.ViewActions.firstVisibleParToAllParIndex(ViewActions.java:284)
	at org.fxmisc.richtext.LineNumberFactory.insertParagraphCheck(LineNumberFactory.java:172)
	at org.fxmisc.richtext.LineNumberFactory.lambda$new$2(LineNumberFactory.java:92)

Add option to strip synthetic flag from code before decompiling

Force methods to be shown even if marked as being compiler generated.

Metadata tracking system for tagging classes and files based on behavior

See: Col-E/Recaf#306

Track metadata for classes and files inside workspaces (class vs file agnostic)

• Can search against metadata (tag search, key-value search, etc)
• Can allow the tree browser to do something like +ui to show only UI classes or -ui to exclude UI classes
• Some metadata can be automatically inferred (See what class extends, how often it references certain types, etc)
• Context action on class/files to open the item's metadata panel where there is full meta access/editing for the selected item

Have a in-memory only MetadataContoller inside the workspace

• Users can define config-persistent rules to auto-tag data
• Recaf will supply a default collection of rules for the JDK classes and how to tag them
• Users can add maven artifacts to rules, ARTIFACT-WIDE or per class.
  • Download it if not in .m2 cache then read it and allow customization of rules

Rules can include:

• Any reference to X
• Parent in hierarchy contains X
• N count of T tagged references
• P percent of tagged items match T tag
• ContainsString S with matchmode M
• >= file size S
• <= file size S
• == file size S
• likely obfuscated

Rule matches can:

• Be tagged with T
• Removed from workspace

Concerns:

• Tagging the current resources must not interfere with rules, so if java.net.Downloader is tagged T and we tag an incoming class Example as T this should not affect N count of T sorts of rules.

Recreate contributors page

Just an alphabetical list of usernames, and links to their github profiles will do.

Improved find in text search

Default bind Control + F

Pressing enter with a search term will go to the next item, wrapping around if necessary.

If there is some selected text in the editor when Control + F is pressed, that text will become the search text. So pressing enter will jump to the next instance of that text.

Disassembler can add descriptive comments to output

For try-catch ranges rather than attempt to name labels we can add trailing comments.

TRY A B CATCH(java/lang/Exception) C
A: // start try-catch
   foo
B: // end try-catch
    bar
C: // start catch(Exception)
    bizz

Migrate to JavaFX 17

Noted from: https://twitter.com/johanvos/status/1436265556244566016

The artifact naming convention has changed.

Some starting points for reference:

• https://github.com/openjdk/jfx/blob/master/build.gradle#L298
• https://repo1.maven.org/maven2/org/openjfx/javafx-controls/16/ vs https://repo1.maven.org/maven2/org/openjfx/javafx-controls/17/

Adding two jars with the same class names causes collision conflicts

Adding two jars (eg obfuscated and non-obfuscated) with the same classpath causes conflicts of the sort:

• Opening a class always opens the one of the first jar imported, never the second
• Cannot open both classes with the same name side by side
• Everything just dicks itself

Ability to duplicate a tab

The tab should include a context menu allowing the duplication of its content into a new tab. This will be useful if the user wants to use the docking framework to display the same class in two different representations at the same time.

I think this will be able to be done entirely in RecafDockingManager

Though this will add some complexity:

• How do we efficiently track a one-to-many for tabs?
  • Goto def, which one is used?

Create a better hex editor

The scroll bar is hidden in the 2x implementation because it makes synchronization of the three columns easier.

This demo may lend to a better version that supports a scrollbar: https://github.com/dlemmermann/JavaFXTipsAndTricks/tree/master/JavaOne2016TipsAndTricks/src/com/dlsc/javaone2016/tips/scrolling

Plus the whole table row design just feels hacky. There must be a better way to do this. If not, at least using Flowless for a more optimized table would be better.

Also test out and compare vs https://github.com/helios-decompiler/hex-editor/tree/master/src/main/java/com/heliosdecompiler/hexeditor

Always keep main section of initial UI state available for tabs

Consider Visual Studio Code. When you open a files and close them the primary space is still always available.

Recaf 3X doesn't do this, so if you close the tabs of the primary space you're left with an awkward vertical split which takes up the full horizontal space.

---

Side note: A lot of the tab drag-drop/layout ought to be refactored to improve UX and internal code cleanliness...

Multithread rename operations to make UI respond quicker

Situation: User opens a jar with 200 classes, 10 of which are opened in decompile tabs. The user renames one class, which is referenced in 30 classes.

The ideal solution:

• The current class is updated, UI immediately refreshes
• On a background thread:
  • The remaining classes are iterated and remapped if they have a reference to the class
  • Each class is updated in the workspace on its own time, but since the user is focusing on the current class these can be done later/in the bg without the user noticing

Offloading the remapping of classes that the user isn't actually looking at means the UI only needs to wait for the current class to be updated, rather than all classes in the worskpace, before it refreshes with the new mapped class.

Call graph generation

Recycle the inheritance graph logic to support generating program flow graphs.

Reflection can be ignored for now, as that is a relatively complex ask.

Escape certain characters when shown in the UI

Should be escaped or hidden, depending on the context. In the members menu, it'd make sense to replace them with a \n that is a different color to denote it being an escaped character. That or removing them entirely.

But if we want to escape references (Blanket replacement is a bad idea) in the decompiled code, we may want to replace with \uXXXX patterns since that is compiler compliant.

Allow users to associate file extensions with supported languages.

When viewing a non-Java file with an extension not matched by default, allow users to choose which language to highlight the file with.

In the case of an unrecognized extension which denotes something that is supported allowing users to say "Hey this is actually XML" would be handy and make reading these files easier. The extension mapping should be remembered as a setting value and be changeable at any time. Maybe have a config page dedicated to discovered extensions?

Add access to the graph based hierarchy panel somewhere

Allowing a quick visual on an entire hierarchy would be cool. For optimization purposes, the node graphs should be cached when generated so re-visiting the same item pulls up the cached graph.

Also updating the graph library so we can further style the graph (like make edges white instead of black) would be useful

Old image of the panel:

New image:

Notify users of unparsable decompilations

If a class is decompiled and JavaParser cannot parse it (meaning right-click interactions won't be supported) the user should be told that the class could not be fully parsed.

Options:

• Switch decompilers
• Use class overview on side panel to interact with fields/methods

Have some placeholder animation on initial decompile

Similar to #25 but, having some fade in for initial decompile would be nice.

Currently, line 1 is empty, then the text appears all at once.

Filter out massive jars from compile classpath

If Recaf opens a ZIP bomb and then tries to compile, it'll run out of heap space trying to parse the ZIP bomb jar.

1. Add filter checks to classpath entries limiting their size
2. Prevent writing a damn zip bomb to the temp classpath dir

Update the info interfaces to support attaching file-based metadata

A system like this could be used for the following issues:

• #2

• Col-E/Recaf#401

When info values are updated in the ClassMap / FileMap types, the old entries metadata should be copied over to the new entry.
The tricky part will be handling remove(oldName) --> put(newName) operations. Will probably just make a utility method for this in the map classes, and remove direct calls to remove/put when the file is being renamed/moved.

Copy over the bug fix for running as an agent on Java 16

System.setProperty("binary.css", "false"); when run as an agent (loaded under classloader in unnamed module) and the JFX thread context classloader setter.

Syncronizing scroll positions after edits is inconsistent

Most of it is hacky wait 300ms then scroll

Instead we should have a pending actions queue and when the representation is ready to display the class then we can process the queued actions. This should make scrolling to items instant as soon as they're ready which should reduce the visual jank described in #25

The ClassRepresentation and FileRepresentation implementations should implement this behavior.

Gracefully handle some of the many unchecked exceptions types ASM likes to throw

Invalid type descriptor? That's an IllegalArgumentException thrown not documented in the javadocs... 🙄

This causes some UI's to bug out if this isn't handled and an exception kills the UI thread stack.

Bozar stupidly long class name does not get removed from tree when class is deleted

Deleting the directory doesn't work because the directory structure is done via string.split('/') but splitting on / doesn't work since there is no actual class name. It ends in / so the tree item lookup fails.

https://github.com/vimasig/Bozar/blob/master/src/main/java/io/github/vimasig/bozar/obfuscator/transformer/impl/CrasherTransformer.java

Consistency in Pane vs Panel naming

Should use Pane since that's the JavaFX convention (Like GridPane/BorderPane)

Closing the [TabPane] that houses the most recent tab prevents new tabs from being opened

1. Open new tab
2. Close the tab and anything in the containing TabPane
3. Try and open a new tab (double click on class/file) and it won't open since the target TabPane for new tabs cannot be found

Opening a tab that already exists should visually make it "pop" to get user attention

Steps

1. User opens class Dog and it appears in a new tab
2. User opens class Cat and it appears in a new tab
3. User opens class Dog and the TabPane containing both [Dog, Cat] switches to Dog, not opening a new instance of Dog
4. The border of the panel representing the Dog class should "pop" to get the user's attention

But what if a user wants to have Dog open twice?

I think having a right-click context menu on the tab title to duplicate the current display would be useful. This will be elaborated on in another issue: #5

Add bypass for fake initial PK file header tricks

So the JVM will look for the LAST occurance of PK.. to start parsing from. But the java.util.zip code does not. So if there is literal garbage after the PK.. at index 0, then later a valid PK.. it refuses to open.

The fix is to remove bullshit data until one of the PK.. and following data is parsable.

Add tooltips for relevant items in PE viewer

The PE viewer only can make so much use of screen real estate. It may be worthwhile to add additional tooltips to certain items to further elaborate on values. Not super important since users can just google the stuff too and read MSDN.

Change hierarchy view to use buttons instead of dropdown menus

Will make it quicker to switch between what is shown.