jtesta / rainbowcrackalack Goto Github PK

Continuous integration should be enabled on this repo to automatically detect problems. Bonus points if the unit tests can be run too!

If there's going to be a bounty for OpenCL/FPGA performance increases, there should be a functioning benchmark mode that doesn't also involve writing result data to disk.

This may be something I'm willing to help with. What exactly was the issue leading to the benchmark mode being disabled?

Its a bit unclear how to generate a full set of tables.
would be nice if you could add the commands for generating full tables or a script for
8-8
9-9
8-9
7-9

I got a minig rig and would support the project if i would know how to generate those tables
if i use crackalack_gen ntlm ascii-32-95 8 8 0 803000 67108864 0
It just generates one file / table ? i guess the command is wrong or some others are missing ?

would be nice if you could post those infos in more detail on the readme

thanks

The lookup code currently does not do any GWS optimization. See:

Hi,

I compiled the code for a windows machine.

I got the following error message if I want to lookup for hashes.

Binary searching will be done with 8 threads.
Loaded 18 of 18 uncracked hashes from hashes.txt.
Pre-computing hash #1: 3df1fd8f0d9c177bec1f6ff3673a26c7...

Note: optimized NTLM8 kernel will be used for precomputation.

Failed to open kernel.: No such file or directory

What files are needed here?? Thank you very much

Any chance of LM/NetNTLMv1 rainbow table creation in future?
Would be a huge help if possible.

Required Ability to restore state of the rainbow_lookup from the moment of interruption.

Why can't the GPU be fully utilized when exploding the rainbow meter?
./crackalack_lookup /tmp/rainbowcrackalack/9_rt c8d9050093b615aa95b73cbd5ae63772

But we can make full use of GPU when generating rainbow table
./crackalack_gen ntlm ascii-32-95 9 9 0 803000 67108864 0

I was wondering if there are any plans to support multiple algorithms for both generation and lookup.
Specifically towards MD5, SHA1, SHA2, and SHA512.
Wouldn't expect the generation of 9 length ascii tables for these of course, just the tools to generate keyspaces & lookups basically

The global work size (GWS) parameter in OpenCL is used to tell a device how many pieces of work to do at a time. Tuning this parameter can result in big improvements in throughput (sometimes over 50%).

Currently, the optimal GWS for each GPU model is determined through manual experimentation and put into gws.c. This method does not scale well, as it leaves out many popular hardware models. A much better method is to add an auto-tuner that determines the optimal setting at run-time.

A proposed solution is this: each time the generation or lookup code is run, it will check if an optimal setting is already known from a previous invokation. This will be done with the following values as a unique key in a hash table: table parameters, device name, driver version (note that the table parameters have been noted to make a difference in optimal GWS; furthermore, driver improvements can make a difference as well). If an optimal setting is already known, it is used; otherwise, variations of the GWS will be tested until an optimal value is found.

The manual GWS command line argument ("-gws") must be preserved in case the user wishes to override this setting.

Currently, the binary search during table lookups is done in the CPU. Investigate if doing this in the GPU is faster, and if so, update the lookup code to do that instead.

When a single hash is given on the command line and it is cracked, the program does not halt immediately. It continues searching tables.

The speed of generating an 8 char rainbow table

The speed of generating an 9 char rainbow table

As mentioned in #1 and this Reddit comment, there is a bias in the implementation of the index_to_plaintext_ntlm9() function in CL/ntlm9_functions.cl.

Trying to run your program under newest Windows 10 machine. Downloaded the 3,5 GB torrent and checked the MD5SUM - all correct. But When I try to run the app it says something like this:

PS D:\hg\Rainbow Crackalack v1.0> .\crackalack_lookup.exe .\rt_ntlm\  D:\hg\Hashes\nthashes.txt

Rainbow Crackalack v1.0
Copyright 2018-2019 Positron Security LLC <https://www.positronsecurity.com/>
Make Rainbow Tables Great Again


Found 2 platforms.
Found 1 devices on platform #0.
Found 2 devices on platform #1.
Device #0:
        Vendor: NVIDIA Corporation
        Name: GeForce GTX 1060
        Version: OpenCL 1.2 CUDA
        Driver: 430.86
        Max compute units: 10
        Max work group size: 1024
        Global memory size: 3221225472

Device #1:
        Vendor: Intel(R) Corporation
        Name: Intel(R) HD Graphics 530
        Version: OpenCL 2.0
        Driver: 22.20.16.4749
        Max compute units: 24
        Max work group size: 256
        Global memory size: 3374045594

Binary searching will be done with 8 threads.
Loaded 44881 of 44881 uncracked hashes from D:\hg\Hashes\nthashes.txt. Failed to infer rainbow table parameters from files in directory.  Ensure that valid rainbow table files are in .\rt_ntlm\ (and/or its sub-directories).
PS D:\hg\Rainbow Crackalack v1.0>

File in folder:
PS D:\hg\Rainbow Crackalack v1.0\rt_ntlm> ls

Directory: D:\hg\Rainbow Crackalack v1.0\rt_ntlm

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----       13.06.2019     19:44           1090 ntlm_loweralpha-numeric-space#1-8_0.md5sums
-a----       13.06.2019     19:56      152302846 ntlm_loweralpha-numeric-space#1-8_0_10000x24663209_distrrtgen[p][i]_9.rti2
-a----       13.06.2019     19:57      414413957 ntlm_loweralpha-numeric-space#1-8_0_10000x67108864_distrrtgen[p][i]_0.rti2
-a----       13.06.2019     19:57      414413911 ntlm_loweralpha-numeric-space#1-8_0_10000x67108864_distrrtgen[p][i]_1.rti2
-a----       13.06.2019     19:57      414412536 ntlm_loweralpha-numeric-space#1-8_0_10000x67108864_distrrtgen[p][i]_2.rti2
-a----       13.06.2019     19:57      414414591 ntlm_loweralpha-numeric-space#1-8_0_10000x67108864_distrrtgen[p][i]_3.rti2
-a----       13.06.2019     19:56      414414823 ntlm_loweralpha-numeric-space#1-8_0_10000x67108864_distrrtgen[p][i]_4.rti2
-a----       13.06.2019     19:57      414414054 ntlm_loweralpha-numeric-space#1-8_0_10000x67108864_distrrtgen[p][i]_5.rti2
-a----       13.06.2019     19:57      414414561 ntlm_loweralpha-numeric-space#1-8_0_10000x67108864_distrrtgen[p][i]_6.rti2
-a----       13.06.2019     19:57      414414181 ntlm_loweralpha-numeric-space#1-8_0_10000x67108864_distrrtgen[p][i]_7.rti2
-a----       13.06.2019     19:57      414416139 ntlm_loweralpha-numeric-space#1-8_0_10000x67108864_distrrtgen[p][i]_8.rti2

Hello
I'm Kim Doo-soo from Korea University in south Korea.
I'm curious about the theoretical background of choosing m corresponding to the number of chains and t corresponding to the chain length in the rainbow table.
For example, I wonder why m=67108864 and t=803000 are 9 digits of the password in this source code.
Please reply. Thank you.

In hash_to_index_ntlm9() it is modulo 6634204312890625 which is 95^8 not 95^9:

Also index_to_plaintext_ntlm9() assumes it's a full 64 bit int:

Leaving index_to_plaintext_ntlm9() unchanged will cause a high bias in the passwords. Since the first 66 characters in the character set have a probability of 1.171875% (3/256) each vs the last 29 characters have 0.78125% (2/256) each. This will cause more collisions and lower the success rate.

I thought we can use CPU instead of GPU.
Can it run without GPU ?

Hi,
First thanks you for this great open source rainbow table software working on GPU !
That's nice !
I'm just starting using it and a bit confused:

• I've created a rainbow table using command: crackalack_gen ntlm ascii-32-95 8 8 0 422000 67108864 X
  • I do not understand how you choose the values 422000, 67108864 and X and what they really mean
• Produced table is named ntlm_ascii-32-95#8-8_0_422000x67108864_0.rt and is near 1Go on disk
  • I've moved it under rt folder
• I've then generated a few passwords using python ./scripts/create_ntlm_passwords.py 8 10
  • This produced 2 passwords files that I clearly understand :)
• I've tried to run ./crackalack_lookup rt/ random_ntlm_hashes_8_chars.txt but it complains explaining this rt//ntlm_ascii-32-95#8-8_0_422000x67108864_0.rt is not a valid table suitable for lookups! (Hint: it may not be sorted.)
• So I've tried to sort the table using ./scripts/rt_sort.sh rt rts
  • But file rainbowcrack.rtsort is not found

Where this file should came from?
Could you provide explanation about the usage of the software so I could contribute by writing a documentation in return?

Thanks a lot

Gaël,

Hello Joe,

I would happilly volunteer a bit by generating some tables but I would like to know how could we help. I generated the first table given in your example but I guess I'm not the first one to generate this one :)

Could we setup a list of already generated tables and the ones missing for the community ?

Regards;

On Windows, looking up a file of hashes doesn't work. This may be because the line endings are CRLF, and the code expects strlen(line) to be 32 instead of 33.

When a single hash is given on the command line, the pot file seems to not be checked first before starting the precomputation.

Users seem to be doing lookups on thousands of hashes. A warning should be issued when more than, say, 2,000 hashes are given to the program, as the time to perform the pre-computations may eclipse the time hashcat would complete.

Got 6x AMD RAD RX 580 8GB
Would like to give them the perfect setting , any advice ?
How to get the best GWS setting for own card ?

Table generation can (in theory) be interrupted and resumed. The code was functioning in early alpha releases, but has not since been tested. A warning is issued to the user upon resumption that it is still experimental and may result in wasted effort (as the resulting table may end up corrupt).

Testing should be done on the resumption functionality to ensure that it is of production quality. If possible, a test should be made in crackalack_tests.py. Lastly, the warning should be removed.

I have a question about what is the max table part index for the NTLM 9-character tables, may you help me

The table lookup processing time can be greatly improved by pre-loading the next table into memory while the current table is being analyzed.

During a test of the NTLM 8 tables, each table was loaded from a solid-state disk in 2.0 seconds. As there are 745 tables in that set, total disk time was 2.0 x 745 = 1490 seconds = almost 25 minutes. This accounts for over 21% of the total run time of the lookup (117 minutes).

The speed-up will be even more dramatic if tables are loaded from a magnetic disk (which will be commonly used when tables greater than 1TB become available).

Hi,
I've downloaded latest (1.2) version and I tried to compile on a centOS machine.
First error is about CL/cl.h missing file.
Any suggestion?
Many thanks

On a system with 2 GPUs, (1xGTX1070 and 1xGTX1080) I get the following when I try to run a lookup.
Two things:

1. The detection finds 2 platforms - and lists each card twice
2. After the initial pre-computing, it just goes back to the initial prompt

D:\Rainbow_Tables\RC13>.\crackalack_lookup.exe d:\Rainbow_Tables\NTLM8 3B1B47E42E0463276E3DED6CEF349F93

Rainbow Crackalack v1.3-dev
Copyright 2018-2020 Positron Security LLC https://www.positronsecurity.com/
Make Rainbow Tables Great Again

Operating system: Windows
Found 2 platforms.
Found 2 devices on platform #0.
Found 4 devices on platform #1.
Device #0:
Vendor: NVIDIA Corporation
Name: GeForce GTX 1080
Version: OpenCL 1.2 CUDA
Driver: 461.40
Max compute units: 20
Max work group size: 1024
Global memory size: 8589934592

Device #1:
Vendor: NVIDIA Corporation
Name: GeForce GTX 1070
Version: OpenCL 1.2 CUDA
Driver: 461.40
Max compute units: 15
Max work group size: 1024
Global memory size: 8589934592

Device #2:
Vendor: NVIDIA Corporation
Name: GeForce GTX 1080
Version: OpenCL 1.2 CUDA
Driver: 461.40
Max compute units: 20
Max work group size: 1024
Global memory size: 8589934592

Device #3:
Vendor: NVIDIA Corporation
Name: GeForce GTX 1070
Version: OpenCL 1.2 CUDA
Driver: 461.40
Max compute units: 15
Max work group size: 1024
Global memory size: 8589934592

Binary searching will be done with 24 threads.
Pre-computing hash #1: 3b1b47e42e0463276e3ded6cef349f93...

Note: optimized NTLM8 kernel will be used for precomputation.

D:\Rainbow_Tables\RC13>

`rainbowcrackalack]$ make test
./crackalack_unit_tests

Rainbow Crackalack v1.2
Copyright 2018-2020 Positron Security LLC https://www.positronsecurity.com/
Make Rainbow Tables Great Again

Operating system: Linux
Found 1 platforms.
Found 2 devices on platform #0.
Device #0:
Vendor: Advanced Micro Devices, Inc.
Name: gfx906
Version: OpenCL 2.0 AMD-APP (3075.10)
Driver: 3075.10 (PAL,HSAIL)
Max compute units: 60
Max work group size: 256
Global memory size: 17163091968

Device #1:
Vendor: Advanced Micro Devices, Inc.
Name: gfx906
Version: OpenCL 2.0 AMD-APP (3075.10)
Driver: 3075.10 (PAL,HSAIL)
Max compute units: 60
Max work group size: 256
Global memory size: 17163091968

Running NTLM index_to_plaintext() tests... passed.
Running NTLM9 index_to_plaintext_ntlm9() tests... passed.
Running NTLM hash tests... Error in hsa_operand section, at offset 1472:
Address offset exceeds variable size
LLVM ERROR:
Brig container validation has failed in BRIGAsmPrinter.cpp

make: *** [Makefile:76: test] Error 1
`
System: Arch linux 5.6.15-arch1-1 #1 SMP PREEMPT Wed, 27 May 2020 23:42:26 +0000 x86_64 GNU/Linux
opencl-headers 2:2.2.20170516-3

Intel GPUs should be tested... especially when mixed with AMD/NVIDIA. The generation code should be able to handle it correctly, but the lookup code may (or may not) be hampered by the slower hardware.

This should be investigated more closely.

Hi, I've the following two Devices being seen in my laptop:

Device #0:
Vendor: NVIDIA Corporation
Name: GeForce GTX 1650
Version: OpenCL 1.2 CUDA
Driver: 419.71
Max compute units: 16
Max work group size: 1024
Global memory size: 4294967296

Device #1:
Vendor: Intel(R) Corporation
Name: Intel(R) UHD Graphics 630
Version: OpenCL 2.1 NEO
Driver: 26.20.100.7985
Max compute units: 24
Max work group size: 256
Global memory size: 13626322944

When I run crackalack_lookup over a file containing multiple hashes, each precomutation it reports to takes 3 minutes.

Looking at my Task Manager I can see that my Internal Graphics card "Intel(R) UHD Graphics 630" is maxed out at 100%, with blips down between precomputations, while my GTX 1650 is sitting idle.

The same appears to happen after precomputation, when I use just one hash and it's searching the tables the internal GPU is spiking regularly, which the GTX 1650 appears not to be doing anything.

Any help to get the program to favor the better of the two graphic devices would be appreciated. Thanks in advance.