lazza / recuperabit Goto Github PK
View Code? Open in Web Editor NEWA tool for forensic file system reconstruction.
License: GNU General Public License v3.0
A tool for forensic file system reconstruction.
License: GNU General Public License v3.0
I started a scanning an image. The size of image is 465 GB.
I used the latest version of recuperabit.
This is the output:
INFO:root:Found NTFS index record at sector 975881640
INFO:root:Found NTFS index record at sector 976504280
INFO:root:Found NTFS index record at sector 976668320
INFO:root:Found NTFS boot sector at sector 976769023
INFO:root:First scan completed
INFO:root:Saving results to settInteressanti.save
INFO:root:Parsing MFT entries
INFO:root:Parsing INDX records
INFO:root:Reading boot sectors
DEBUG:root:Dropping bogus NTFS partition with MFT position 2064 generated by MFT mirror of partition at offset 2048
DEBUG:root:Dropping bogus NTFS partition with MFT position 86432682 generated by MFT mirror of partition at offset 86427520
DEBUG:root:Dropping bogus NTFS partition with MFT position 128554554 generated by MFT mirror of partition at offset 128549392
DEBUG:root:Dropping bogus NTFS partition with MFT position 372031174 generated by MFT mirror of partition at offset 372026012
Traceback (most recent call last):
File "/opt/RecuperaBit/main.py", line 328, in <module>
main()
File "/opt/RecuperaBit/main.py", line 310, in main
parts.update(scanner.get_partitions())
File "/opt/RecuperaBit/recuperabit/fs/ntfs.py", line 798, in get_partitions
relative = datas[0]['runlist'][0]['offset']
IndexError: list index out of range
xubuntu@xubuntu-virtual-machine:/media/xubuntu/Maxtor 1.3 GB$
Hi,
First, thank you, this is a great tool. The ability to restore filenames and directories really helps in tracking down lost data. This may be more of an enhancement request, but I think the ability to restore the original file's creation/modification date would further assist in investigations.
I was running RecuperaBit on a 466GB image for a little longer than 100h. I looked at the output and it was repeating the same positions set number.
For example, if you look at the output file, look at the first three positions: 38190544, 37475152, 38357392. The last three positions on the end are the same: 38190544, 37475152, 38357392.
I was able to grab only this part of the output, but when I checked there was more repetition (around 10 or 15 positions). I realized it was repeating the same sequence over and over.
The drive I'm running at is a very damaged one. I'm not sure if it's the image issue (unrecoverable) and I should let it go.
(some more info: running on Mac OS 10.12.1, was using a whole lot of memory, around 12GB~14GB)
Python 2.7.12 (aff251e543859ce4508159dd9f1a82a2f553de00, Nov 13 2016, 01:57:41)
[PyPy 5.6.0 with GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
It appears that if you have a drive with a lot of files, when it goes to recreate the MFT, it runs out of memory.
I have a 4TB drive that had only 1 partition, but (rough guess) had hundreds of thousands of files.
Is there any debugging that I can add to try and figure out how much RAM will be needed? I have access to a box with 192gigs of RAM, but I would really like to figure out how much RAM will be needed for it to complete before I requisition the box.
Hi I had a similar problem then what is listed here: http://stackoverflow.com/questions/37539950/corrupt-ntfs-folder-not-accessible-from-either-windows-mas-osx-or-linux
The hard drive that had the problem is an ssd, and it had my windows install on it. I can see all folders when i mount the drive with the exception of 2, the "c:\Users" being one of them. I created a partition and installed linux on another HD.
Since the drive doesn't seem to be damaged i ran the application directly on it. The first time it ran it crashed midway, i don't know what happened the terminal just froze, and no savefile was created. I ran the application 4 times since then, the first one took a long time (around 20 hours) and created a savefile. The other other ones ran much faster around 2 hours. But they all end up the same with the message:
INFO:root:Found NTFS boot sector at sector 234229752
INFO:root:First scan completed
INFO:root:Parsing MFT entries
INFO:root:Parsing INDX records
INFO:root:Reading boot sectors
DEBUG:root:Dropping bogus NTFS partition with MFT position 27585690 generated by MFT mirror of partition at offset 27580528
DEBUG:root:Dropping bogus NTFS partition with MFT position 65746898 generated by MFT mirror of partition at offset 65741736
DEBUG:root:Dropping bogus NTFS partition with MFT position 112 generated by MFT mirror of partition at offset 0
INFO:root:Finding partition geometry
INFO:root:Finalizing MFT reconstruction of partition at offset 27580528
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 65741736
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 0
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
But then a series of messages appear saying:
WARNING:root:Cannot read sector(s). Filling with 0x00. Offset: 119785142 Size: 2 Bsize: 512
After a bunch of these, the folowing apppears:
Traceback (most recent call last):
File "main.py", line 328, in <module>
main()
File "main.py", line 310, in main
parts.update(scanner.get_partitions())
File "/home/leandro/RecuperaBit-master/recuperabit/fs/ntfs.py", line 841, in get_partitions
self.finalize_reconstruction(part)
File "/home/leandro/RecuperaBit-master/recuperabit/fs/ntfs.py", line 674, in finalize_reconstruction
self.add_from_attribute_list(parsed, part, node.offset)
File "/home/leandro/RecuperaBit-master/recuperabit/fs/ntfs.py", line 629, in add_from_attribute_list
_integrate_attribute_list(parsed, part, image)
File "/home/leandro/RecuperaBit-master/recuperabit/fs/ntfs.py", line 221, in _integrate_attribute_list
entries += attribute_list_parser(dump)
File "/home/leandro/RecuperaBit-master/recuperabit/fs/ntfs_fmt.py", line 118, in attribute_list_parser
content.append(decoded)
MemoryError
Now I'm lost on what to do, and what could be causing the problem. And when I try to run the "recoverable" command it just says "command not found", I'm assuming because the program crashed...
Okay so I'm trying to recover files from a Windows 10 partition on a 1TB WD blue hard drive. After running the command "recuperabit /dev/sdb3 -o Recovery -s Windows10.save" it asks me to proceed so I say yes, and it begins the scanning process but then after sometime of scanning, it seems like it all just comes to a halt and never continues. It doesn’t prompt me to enter another command, the scan just stops with no prompt. If I do try to enter another command like "recover", nothing happens. Please help me figure out what to do. I absolutely need to recover some files.
Hi,
Was trying to use your application to recover deleted files/directories on a 320GB hard disk. When i ran the main.py command it output with the below error. Is there something i am missing?
DEBUG:root:Found MATCH in positions set([30803328]) with weight 6330 (73.6132108
385%)
INFO:root:Finalizing MFT reconstruction of partition at offset 30801920
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
ERROR:root:Cannot handle multiple attribute $VOLUME_INFORMATION
Traceback (most recent call last):
File "./RecuperaBit-master/main.py", line 385, in <module>
main()
File "./RecuperaBit-master/main.py", line 367, in main
parts.update(scanner.get_partitions())
File "/media/ubuntu/d807d211-57d8-47be-8296-72bd6bba7d0d/paula/RecuperaBit-mas
ter/recuperabit/fs/ntfs.py", line 847, in get_partitions
self.finalize_reconstruction(part)
File "/media/ubuntu/d807d211-57d8-47be-8296-72bd6bba7d0d/paula/RecuperaBit-mas
ter/recuperabit/fs/ntfs.py", line 677, in finalize_reconstruction
self.add_from_attribute_list(parsed, part, node.offset)
File "/media/ubuntu/d807d211-57d8-47be-8296-72bd6bba7d0d/paula/RecuperaBit-mas
ter/recuperabit/fs/ntfs.py", line 632, in add_from_attribute_list
_integrate_attribute_list(parsed, part, image)
File "/media/ubuntu/d807d211-57d8-47be-8296-72bd6bba7d0d/paula/RecuperaBit-mas
ter/recuperabit/fs/ntfs.py", line 250, in _integrate_attribute_list
child_parsed = parse_file_record(dump)
File "/media/ubuntu/d807d211-57d8-47be-8296-72bd6bba7d0d/paula/RecuperaBit-mas
ter/recuperabit/fs/ntfs.py", line 150, in parse_file_record
attributes = _attributes_reader(entry, header['off_first'])
File "/media/ubuntu/d807d211-57d8-47be-8296-72bd6bba7d0d/paula/RecuperaBit-mas
ter/recuperabit/fs/ntfs.py", line 132, in _attributes_reader
raise NotImplementedError
NotImplementedError
A way to pause and resume scanning of big drives would be a cool feature.
I've started RecuperaBit on a USB3 8TB drive (that i've overwritten the first 1-2GB) and it's taking a while. So far it's about 2/3 done but it's still going to be another day or two until it finishes. Would not be a problem usually but i've stupidly forgot to put it in a screen and now i'm stuck waiting for it to finish :/
I have a 3TB drive that I'm currently recovering files from. I can get the list of files nice and easy with the analysis pass so it would be easy to just use restore x 5 and restore x -1 to recover everything. However, it seems that some previously deleted files (marked as deleted in the csv output) are being recovered as well, and this means that I run out of disk space if I use the simple route (I don't have a 4TB drive to recover to).
In my case, I don't need those deleted files as they are either from the Windows Recycle Bin or directly deleted by me or some program. It would be nice if I could use a parameter to skip the deleted files or maybe skip files and/or directories by name using a wildcard.
I've started with an external 1TB HDD (msdos partition table) with a single NTFS partition that was broken by partially overwriting the partition by a smaller FAT partition (< 1.3 GB, possibly not fully written, the partition table was amended though).
The RecuperaBit successfully located the partition and most (if not all) of the files that were lost in the process. As a storage to recover the files I used an additional external 3TB HDD (gpt) with a single NTFS partition, already containing some files. I've made a dedicated directory for the restored files and proceeded with the restore.
For the procedure I've used a GNU/Linux system with ntfs-3g
implementation of the NTFS support. After the procedure, I've noticed that the destination directory contained NTFS metafiles that appeared as a normal files ($Mft
, $Secure
, ...) . Not sure if the permissions of the recovered files were valid either, but the system was able to open them, so I didn't really care, I assume that because of a different color in the terminal listing.
I've decided to delete those metafiles because I thought they are irrelevant for the new filesystem, and after unmounting it I'm no longer able to mount it again. Windows freezes or is stuck in a connect – disconnect loop, GNU/Linux fails with the following output:
ntfs_mst_post_read_fixup_warn: magic: 0xff00ffff size: 4096 usa_ofs: 65535 usa_count: 65535: Invalid argument
Actual VCN (0xffffffffffffffff) of index buffer is different from expected VCN (0x0).
Failed to open $Secure: No such file or directory
So, IMO, RecuperaBit should not try to write metafiles if the target file system is ntfs and/or should use a valid name for them. The "legit" metafiles should be created by ntfs-3g
when restored files are written. I'm assuming RecuperaBit uses standard python IO for creating files.
It is also possible that ntfs-3g
should refuse to write those files in the target file system.
As a side note, any ideas on how to approach fixing the target NTFS now are welcome 🙂 .
I have an NTFS partition that got screwed up through chkdsk
and ntfsfix
, but any partition is too small to store an IMG. Why can't I refer to /dev/sda4
? What could happen?
Hi,
I am trying to recover an NTFS formatted disk (for your information there is an underlying RAID 5 (3 disks + 1 spare)) whose partition informations are lost (even MFT and MFT mirror are lost!). I have tried the most updated RecuperaBit (1.1.1), launching the command:
recuperabit -s /mnt/tera8/atpmicrobusrecovery/savefile -o /mnt/tera8/atpmicrobusrecovery /dev/cciss/c0d0
and obtaining:
[.....omissis.....]
INFO:root:First scan completed
INFO:root:Parsing MFT entries
Traceback (most recent call last):
File "/usr/local/bin/recuperabit", line 358, in <module>
main()
File "/usr/local/bin/recuperabit", line 340, in main
parts.update(scanner.get_partitions())
File "/mnt/tera8/rb/RecuperaBit/recuperabit/fs/ntfs.py", line 722, in get_partitions
part.add_file(NTFSFile(parsed, position, ads=ads_name))
File "/mnt/tera8/rb/RecuperaBit/recuperabit/fs/ntfs.py", line 270, in __init__
ads_suffix = ':' + ads if ads != '' else ads
TypeError: cannot concatenate 'str' and 'NoneType' objects
Any idea?
Thanks.
Currently, RecuperaBit does not verify if the output directory specified via -o
exists or if it can be created. The program should at least check its existence and create it if needed.
Hi there, I'm trying to recover an external 4tb WD My Book drive. on which I accidentally used Startup Disk Creator to create an Ubuntu startup disk, so I think the first 3 or 4 gigs may be overwritten. so I tried recuperabit and it works fine until parsing the INDX records, then the program quits with next error:
INFO:root:Found NTFS file record at sector 3906986000
INFO:root:Found NTFS boot sector at sector 7813969912
INFO:root:First scan completed
INFO:root:Saving results to uitvoer.save
INFO:root:Parsing MFT entries
INFO:root:Parsing INDX records
Traceback (most recent call last):
File "/usr/local/bin/recuperabit", line 385, in <module>
main()
File "/usr/local/bin/recuperabit", line 367, in main
parts.update(scanner.get_partitions())
File "/home/henri/RecuperaBit/recuperabit/fs/ntfs.py", line 733, in get_partitions
parsed = parse_indx_record(dump)
File "/home/henri/RecuperaBit/recuperabit/fs/ntfs.py", line 182, in parse_indx_record
name_ok = file_name['name'] is not None
UnboundLocalError: local variable 'file_name' referenced before assignment
Please any advice on what might be wrong, and what to do.
I'm using Ubuntu 18.04 LTS
Python 2.7.15rc1 installed
greetings Henri
When trying to recover large drives, it helps to know how much time you need to wait.
For example, photorec
displays this:
Elapsed time 0h00m04s – Estimated time to completion 0h22m54
P.S. Thank you for this tool, looks promising! Although I'm still waiting for it to scan the whole drive…
Edit: It worked! I recovered the files when both testdisk and photorec didn't help at all. Amazing.
I've emailed before about recovering all partitions and I understand that was not a great idea. Now I raise the question, If it is possible to recover the partitions that are listed as recoverable instead of having to individually filter through each one. I am confident that each partition contains important date because as I am looking for a variety of file types on this rescue image. If you could lead me to the solution or a better solutions that would also be appreciated. Thanks you in advance.
I ran into a OOM-kill after which the savefile was broken (unexpected eof), kind of breaks the purpose for having a save-file.
I suggest closing (or flushing) the savefile before continuing to avoid this issue. Diff included.
index 5cea733..e6ed491 100755
--- a/main.py
+++ b/main.py
@@ -333,6 +333,7 @@ def main():
logging.info('Saving results to %s', args.savefile)
savefile = open(args.savefile, 'wb')
pickle.dump(interesting, savefile)
+ savefile.close()
# Ask for partitions
parts = {}```
I've had to completely scan my drive three times today because I didn't specify an output directory at my initial issuance of the command to run the program! This has been an arduous process. The README suggests using parameter -s to load from a savefile but it's unclear on how to create a savefile after a scan in the first place. I'm very glad and thankful that you've put together this program, but I wish the README were more clear on this issue. How can savefiles be created?
Since it is possible to dump csv to a file, why can the tree command only output to stdout? It would be convenient if it was possible to automatically save the output to a file.
I, for example, had quite a large partition with lots of files. It was most convenient for me to copy&paste output of the tree command to a file, then inspect it and delete rows that were of no interest for me, only leaving these files that could be useful; and according to this list I started recovering files.
Dewald and Seufert describe an approach that can be used to reconstruct Ext4 file systems even if the metadata is damaged: Andreas Dewald, Sabine Seufert, AFEIC: Advanced forensic Ext4 inode carving, Digital Investigation, Volume 20, Supplement, March 2017, Pages S83-S91, ISSN 1742-2876.
Their approach seems to have some similarities to the NTFS module of RecuperaBit:
[We] develop a novel approach to identify files in an Ext4 file system even in cases where the superblock is corrupted or overwritten, e.g. because of a re-formatting of the volume. Our approach applies heuristic search patterns for utilizing methods of file carving and combines them with metadata analysis.
The paper (full text) should be carefully studied to verify if this approach can be turned into a RecuperaBit plug-in.
Just a suggestion for a minor improvement: :-)
I'm using RecuperaBit for the first time and right after starting the program I got this prompt:
Type [Enter] to start the analysis or "exit" / "quit" / "q" to quit:
After pressing [Enter] nothing happens on the screen. A little Scanning the image, this might take quite some time...
or anything similar (or even a progress bar) would be nice.
In my case I wondered if RecuperaBit was still waiting for input, had locked up or whatever. After a few minutes I had to check the disk I/O statistics to understand that RecuperaBit was actually reading heavily from the disk (since I use RecuperaBit remotely I can't hear the disks working, so I have absolutely no feedback...).
That said, your program looks very promising. Keep up the good work!
Hey there, I came across the program via your recommendation on SE. As I mentioned on there, I've been trying to recover the directory structure from a 2TB drive for months and have used a ton of programs in the process, including GetDataBack and TestDisk, so will be very impressed if RecuperaBit lives up to the level of capability that your Masters thesis outlines it can.
Unfortunately, I can't seem to get it to work with my drive using Cygwin on Windows 7, even though the RB's documentation states it should work with "real drives". I'm not an expert in bash
and/or Python by any means, so it's possible I'm getting the syntax wrong.
Thanks in advance for your help, I really appreciate it.
so im trying to recover a drive that got erased with hddscan, the erase option got instantly aborted only the first 1mb of the disk shows the hddscan overwrite pattern.
whenever i try to do a tree output or restore recuperabit crashes with the following message:
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 66: ordinal not in range(128)
im using git master, the crash occurs after some seconds after it already restored about 300mb of files
traceback:
INFO:root:Restoring #273517 Root/Program Files/Windows NT/TableTextService/TableTextService.dll
Traceback (most recent call last):
File "/usr/lib/pypy/lib-python/2.7/logging/init.py", line 897, in emit
stream.write(fs % msg.encode("UTF-8"))
File "/usr/lib/pypy/lib-python/2.7/codecs.py", line 369, in write
data, consumed = self.encode(object, self.errors)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 62: ordinal not in range(128)
Logged from file logic.py, line 231
Traceback (most recent call last):
File "main.py", line 385, in
main()
File "main.py", line 382, in main
interpret(cmd, arguments, parts, shorthands, args.outputdir)
File "main.py", line 191, in interpret
logic.recursive_restore(myfile, part, partition_dir)
File "/recupera/recuperabit/logic.py", line 258, in recursive_restore
recursive_restore(child, part, outputdir, make_dirs=False)
File "/recupera/recuperabit/logic.py", line 258, in recursive_restore
recursive_restore(child, part, outputdir, make_dirs=False)
File "/recupera/recuperabit/logic.py", line 258, in recursive_restore
recursive_restore(child, part, outputdir, make_dirs=False)
File "/recupera/recuperabit/logic.py", line 232, in recursive_restore
if not makedirs(restore_path):
File "/recupera/recuperabit/logic.py", line 196, in makedirs
os.makedirs(path)
File "/usr/lib/pypy/lib-python/2.7/os.py", line 157, in makedirs
mkdir(name, mode)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 66: ordinal not in range(128)
I tried using the new merge functionality on the 5 partitions I referenced in issue #18 , however the recovery wasn't successful.
Here is the output:
https://pastebin.com/sns4t84A
Notice first how the root doesn't contain any files. In the individual partitions I had recovered before, some of them had files in the root directory, therefore I would assume that at least those files would end up in the root once I merged these partitions.
As for the LostFiles, it has very few files and none of them were part of the original partition.
In fact from the bodyfile, I noticed that their timestamps were from before the original partition was even created.
Here are the exact commands I executed:
merge 312 451
merge 312 283
merge 312 248
merge 312 1007
Then I tried to restore partition #312
.
After the merge, it was listed like this when issuing the recoverable
command:
Partition #312 -> Partition (NTFS, ??? b, 576538 files, Recoverable, Offset: 264192, Offset (b): 135266304, Sec/Clus: 8, MFT offset: 4287141256, MFT mirror offset: None)