Support of multiple NTUSER.DAT and UsrClass.dat about autoripy

[Silv3rHorn]

Heard good things about regipy as well, but I am more keen to leverage what I previously written 😄 - RegistryFlush (executable available here https://github.com/Silv3rHorn/4n6_misc/releases). Now, I just need to combine them both together. Will try to get this done by the end of the week.

[evild3ad]

I updated my own Artifacts-Collector script and it works with your nomenclature for me now. Thx!

Do you know regipy? Maybe it helps to check out "registry-transaction-logs" command. It is also in Python.
https://github.com/mkorman90/regipy

[CmdrBurrito]

FYI, I've also been manually merging the transaction logs into the various registry hives before processing with "autoripy".

However, I've been doing it with Eric Zimmerman's "rla.exe" CLI tool, along with a (fairly complex) script that orders all of the merged output hive files into a directory structure that can be easily parsed by "autoripy" using the -s, -a, and -m parameters.

I think it would be great if "autoripy" could just look for and automatically merge any transaction log files that happen to be present in the same directories as the .DAT's before processing them, as that would save me from having to perform that extra pre-processing (i.e., manual merge) step.

I will have to take a look at "regipy" now (I had never heard of it before), to see if it's any easier to use than the "rla.exe" tool I've been using to merge the transaction logs with the registry hives.

[CmdrBurrito]

Hi @Silv3rHorn, FYI, I decided to give your new exe version of "autoripy" a shot, but ran into the following error after running it:

C:\IRTools\RegRipper3.0>autoripy-20200909.exe
-rr .
-s D:\Cases\XXXXX\Artifacts\RegFiles
-a D:\Cases\XXXXX\Artifacts\RegFiles
-m D:\Cases\XXXXX\Artifacts\RegFiles\Users
-r D:\Cases\XXXXX\Analysis\All_Users_Autoripy-20200909_RR3.0
--flush

Flush failed (no new data) - D:\Cases\XXXXX\Artifacts\RegFiles\SAM
Flush successful - D:\Cases\XXXXX\Artifacts\RegFiles\SECURITY
Traceback (most recent call last):
File "autoripy.py", line 174, in
File "autoripy.py", line 168, in main
File "autoripy.py", line 140, in run_flush
File "autoripy.py", line 115, in _flush
File "yarp\Registry.py", line 326, in recover_auto
File "yarp\Registry.py", line 266, in recover_new
File "yarp\RegistryFile.py", line 1395, in apply_new_log_files
File "yarp\RegistryFile.py", line 1356, in apply_new_log_file
File "yarp\RegistryFile.py", line 1140, in build_cell_maps
File "yarp\RegistryFile.py", line 1121, in hive_bins
File "yarp\RegistryFile.py", line 555, in init
yarp.RegistryFile.HiveBinException: "Invalid signature: b'\x00\x00\x00\x00'"
[11000] Failed to execute script autoripy

FYI, the registry data for my case is laid out as follows (note: this is post-autoripy execution):

D:\Cases\XXXXX\Artifacts\RegFiles>dir
Volume in drive D is .....
Volume Serial Number is .....

Directory of D:\Cases\XXXXX\Artifacts\RegFiles

09/09/2020 14:46 PM <DIR> .
09/09/2020 14:46 PM <DIR> ..
08/21/2020 16:28 PM 3,670,016 Amcache.hve
08/07/2018 22:24 PM 327,680 Amcache.hve.LOG1
08/07/2018 22:24 PM 114,688 Amcache.hve.LOG2
08/21/2020 16:26 PM 2,621,440 DEFAULT
04/11/2018 21:04 PM 200,704 DEFAULT.LOG1
04/11/2018 21:04 PM 131,072 DEFAULT.LOG2
08/21/2020 15:59 PM 131,072 SAM
04/11/2018 21:04 PM 65,536 SAM.LOG1
04/11/2018 21:04 PM 49,152 SAM.LOG2
09/09/2020 14:46 PM 65,536 SECURITY
04/11/2018 21:04 PM 75,776 SECURITY.LOG1
04/11/2018 21:04 PM 75,776 SECURITY.LOG2
08/21/2020 16:26 PM 65,536 SECURITY.old
08/21/2020 16:26 PM 117,702,656 SOFTWARE
04/11/2018 21:04 PM 27,262,976 SOFTWARE.LOG1
04/11/2018 21:04 PM 25,165,824 SOFTWARE.LOG2
08/21/2020 16:26 PM 22,020,096 SYSTEM
04/11/2018 21:04 PM 5,404,672 SYSTEM.LOG1
04/11/2018 21:04 PM 5,551,104 SYSTEM.LOG2
08/21/2020 22:09 PM <DIR> Users

The autoripy log file from this run is: [email protected]

Also, FYI, this version of autoripy seems to run to completion just fine as long as the new "--flush" command-line option is NOT specified.

---

UPDATE:

See my comment below for additional details regarding this issue...

[evild3ad]

I tested autoripy against a Windows 10 Enterprise (x64), Version: 1809 (10.0.17763.1) without errors.

"Flush successful" when dirty and "Flush failed (no new data)" when not dirty.

My PowerShell command:
$autoripy = "C:\Tools\autoripy\autoripy.exe"
& $autoripy "C:\Tools\RegRipper" -s "$OUTPUT\Registry\Registry" -m "$OUTPUT\Registry\Registry\Users" -r "$OUTPUT\Registry\Analysis\RegRipper" --flush > $null

I will have a deeper look on it on the weekend...but it is already integrated in my Artifacts-Collector for all the next forensic analysis jobs. ;-)

Thank you!

[CmdrBurrito]

BTW, @Silv3rHorn just tested out your stand-alone "registryFlush" tool. OMG, where has that been all my life? It's awesome! Would have saved me a ton of work vs having to script up the "rla.exe" tool I was previously using, and for my purposes it's way easier to use than the aforementioned "regipy" tool as well.

[CmdrBurrito]

Ok, I just figured out what is causing my issue with "autoripy-20200909.exe" (see comment above). Turns out that the "registryFlush" tool/utility errors out whenever it gets to, and attempts to integrate, the transaction logs of the SOFTWARE hive that is being processed from this particular host. Unsure why. FYI, Eric Zimmerman's "rla.exe" tool seems to process this same SOFTWARE hive, and its associated transaction logs, without errors.

Running "registryFlush" stand-alone against the SOFTWARE hive in question yields the error message:

C:\IRTools\Silv3rHorn>registryFlush_20181209.exe -f D:\Cases\XXXXX\Artifacts\RegFiles\SOFTWARE --overwrite
Traceback (most recent call last):
File "registryFlush.py", line 110, in
File "registryFlush.py", line 85, in main
File "site-packages\yarp-1.0.27-py3.7.egg\yarp\Registry.py", line 312, in recover_auto
File "site-packages\yarp-1.0.27-py3.7.egg\yarp\Registry.py", line 252, in recover_new
File "site-packages\yarp-1.0.27-py3.7.egg\yarp\RegistryFile.py", line 1391, in apply_new_log_files
File "site-packages\yarp-1.0.27-py3.7.egg\yarp\RegistryFile.py", line 1352, in apply_new_log_file
File "site-packages\yarp-1.0.27-py3.7.egg\yarp\RegistryFile.py", line 1136, in build_cell_maps
File "site-packages\yarp-1.0.27-py3.7.egg\yarp\RegistryFile.py", line 1117, in hive_bins
File "site-packages\yarp-1.0.27-py3.7.egg\yarp\RegistryFile.py", line 551, in init
yarp.RegistryFile.HiveBinException: "Invalid signature: b'\x00\x00\x00\x00'"
[2588] Failed to execute script registryFlush

[evild3ad]

I used yaru (TZWorks, Full Version) to compare hive.old and recovered hive. All good...the new autoripy version works!

[CmdrBurrito]

Thank you! I can confirm that this newest/latest version of autoripy (i.e.,"autoripy-20200911") is now working as expected (with no crashes) in all cases! Much appreciated!

[Silv3rHorn]

Hi,

Apologies, I missed this as I did not receive an email notification that an issue had been raised.

Support for specified nomenclature - I am not keen to support other nomenclatures as every user might have their own (and I can't support all of them). Would consider if the proposed nomenclature is an industry-standard or supported by many other tools.

Support for transaction logs - That's a good idea. I had always been doing it manually before parsing with autoripy. Will see what I can do.

Regards

[evild3ad]

Sounds good! I'm happy to test the combined version.

[Silv3rHorn]

@evild3ad, since you offered 😄 , would appreciate if you could test this new version and let me know how it goes. It also backups the previous version under the name <hive>.old

[evild3ad]

No prob! ;-)

[evild3ad]

All hive.old have the same file size in KB. It would be nice to see how much data was recovered from which hive in the autoripy log.

[Silv3rHorn]

@evild3ad - Thanks for testing. Pertaining to the suggestion to include the number of recovered pages, yarp library (used to merge transaction logs) does not seem to return that value. I understand that regipy does it, but after some testing, I noticed that the value returned is not the number of pages recovered but the number of pages found in the transaction logs. There are many times when the transaction logs do not contain new data but regripy continues to merge those logs.

@CmdrBurrito, glad that you like registryFlush. It is a wrapper around Maxim Suhanov's yarp library and the wrapper was wrote in the way that I wanted to use it.

As for the error you encountered, unfortunately, there is little I can do about it since the error is at the yarp library side. I have created a new release version that catches such error, skips the flushing, and continue running the tool. I would suggest raising an issue to Maxim to fix it.

[Silv3rHorn]

FYI, I also created a autoripy version using regipy, but during my testing, I realised that it did not merge the transaction logs properly for some of the hives, resulting in missing data in autoripy's output. Hence, I reverted back to the yarp version. I will be raising an issue with regipy with what I had found.

[CmdrBurrito]

Thanks @Silv3rHorn! However, I just tested with the new version of "autoripy" (autoripy-20200910.exe), but am still getting an error when it hits that one SOFTWARE hive and its associated transaction logs:

C:\IRTools\Silv3rHorn>autoripy-20200910.exe -rr C:\IRTools\RegRipper3.0 -s D:\Cases\XXXXX\Artifacts\RegFiles -a D:\Cases\XXXXX\Artifacts\RegFiles -m D:\Cases\XXXXX\Artifacts\RegFiles\Users -r D:\Cases\XXXXX\Analysis\All_Users_Autoripy-20200910_RR3.0 --flush

_Flush failed (no new data) - D:\Cases\XXXXX\Artifacts\RegFiles\SAM
Flush successful - D:\Cases\XXXXX\Artifacts\RegFiles\SECURITY
Traceback (most recent call last):
File "autoripy.py", line 182, in
File "autoripy.py", line 176, in main
File "autoripy.py", line 148, in run_flush
File "autoripy.py", line 123, in flush
ValueError: Single '}' encountered in format string
[7924] Failed to execute script autoripy

[email protected]

Looks like the new "registryFlush" (registryFlush-20200910.exe) still doesn't like that SOFTWARE hives' transaction logs:

C:\IRTools\Silv3rHorn>registryFlush-20200910.exe -f D:\Cases\XXXXX\Artifacts\RegFiles\SOFTWARE --overwrite
Traceback (most recent call last):
File "registryFlush.py", line 115, in
File "registryFlush.py", line 93, in main
ValueError: Single '}' encountered in format string
[3336] Failed to execute script registryFlush

FYI, the new "autoripy" does run through all of the evidence files just fine as long as I remove the troublesome SOFTWARE hive transaction log files from the target directory (or merge them before-hand using the "rla.exe" tool).

Guess it's time for me to follow up with Maxim? Many thanks!

[Silv3rHorn]

🤦‍♂️ , my bad, I left an extra brace in my code and didn't notice it. Was expecting my IDE to catch such errors. I have replaced the 2 release executables with a fix for that. However, pls note that it doesn't successfully "flush" the transaction logs of the SOFTWARE hive that you have issues with, but just skips it when it encounters an error and continues the processing.

[CmdrBurrito]

Much appreciated @Silv3rHorn! Unfortunately, I just downloaded and ran the newly updated/patched version of "autoripy", and am now getting a slightly different error when it attempts to integrate the SOFTWARE hive transaction logs:

C:\IRTools\Silv3rHorn>autoripy-20200910.exe -rr C:\IRTools\RegRipper3.0 -s D:\Cases\XXXXX\Artifacts\RegFiles -a D:\Cases\XXXXX\Artifacts\RegFiles -m D:\Cases\XXXXX\Artifacts\RegFiles\Users -r D:\Cases\XXXXX\Analysis\All_Users_Autoripy-20200910_RR3.0 --flush

Flush failed (no new data) - D:\Cases\XXXXX\Artifacts\RegFiles\SAM
Flush successful - D:\Cases\XXXXX\Artifacts\RegFiles\SECURITY
Flush failed ("Invalid signature: b'\x00\x00\x00\x00'") - D:\Cases\XXXXX\Artifacts\RegFiles\SOFTWARE
Traceback (most recent call last):
File "autoripy.py", line 182, in
File "autoripy.py", line 176, in main
File "autoripy.py", line 148, in run_flush
File "autoripy.py", line 124, in flush
ValueError: Single '}' encountered in format string
[7216] Failed to execute script autoripy

FYI, the generated autoripy log file contents are unchanged from the previously reported unsuccessful run.

Understood that the intended behavior is that it's supposed to just skip the transaction log integration, if it runs into an error during that process. As long as the transaction log merge failure is noted somewhere within the tools output, we're golden. :)

Will have to follow-up with Maxim regarding getting a fix implemented for any actual merge problems.

UPDATE:

Actually, running the newest/latest updated/patched version of "registryFlush" directly against the problematic SOFTWARE hive and its associated transaction logs, doesn't result in a crash anymore. For example:

C:\IRTools\Silv3rHorn>registryFlush-20200910.exe -f D:\Cases\XXXXX\Artifacts\RegFiles\SOFTWARE --overwrite
Flush failed ("Invalid signature: b'\x00\x00\x00\x00'") - D:\Cases\XXXXX\Artifacts\RegFiles\SOFTWARE.LOG2
Time Taken: 0:00:14.325270

As such, this latest "autoripy" issue looks like it might actually be related to another misplaced code brace (or similar) problem?

[Silv3rHorn]

@CmdrBurrito, As you mentioned, I forgot to modify another line. I have now fixed it, reproduced a similar error and made sure it works (see below). Latest release executable is available here.

autoripy C:\git\RegRipper3.0 -s Registry -a MRU\Prog\amcache -m Registry -r autoripy -c all --flush


Flush failed (no new data)    - D:\OneDrive\git\test\mus2019_win10\Registry\SAM
Flush successful              - D:\OneDrive\git\test\mus2019_win10\Registry\SECURITY
Flush failed (HiveBinException: "Invalid signature: b'0000'") - D:\OneDrive\git\test\mus2019_win10\Registry\SOFTWARE
Flush successful              - D:\OneDrive\git\test\mus2019_win10\Registry\SYSTEM
Flush successful              - D:\OneDrive\git\test\mus2019_win10\MRU\Prog\amcache\Amcache.hve
Flush successful              - D:\OneDrive\git\test\mus2019_win10\Registry\Administrator\NTUSER.DAT
Flush failed (no new data)    - D:\OneDrive\git\test\mus2019_win10\Registry\SelmaBouvier\NTUSER.DAT
Flush successful              - D:\OneDrive\git\test\mus2019_win10\Registry\Administrator\UsrClass.DAT
Flush failed (no new data)    - D:\OneDrive\git\test\mus2019_win10\Registry\SelmaBouvier\UsrClass.DAT


---- Processing the COMM category
---- Processing the DEVICE category
---- Processing the MALWARE category
...