guidobartoli / sherloq

An open-source digital image forensic toolset

An open source image forensic toolset

Introduction

"Forensic Image Analysis is the application of image science and domain expertise to interpret the content of an image and/or the image itself in legal matters. Major subdisciplines of Forensic Image Analysis with law enforcement applications include: Photogrammetry, Photographic Comparison, Content Analysis, and Image Authentication." (Scientific Working Group on Imaging Technologies)

Sherloq is a personal research project about implementing a fully integrated environment for digital image forensics. It is not meant as an automatic tool that decide if an image is forged or not (that tool probably will never exist...), but as a companion in experimenting with various algorithms found in the latest research papers and workshops.

While many commercial solutions have high retail prices and often reserved to law enforcement and government agencies only, this toolset aims to be a both an extensible framework and a starting point for anyone interested in making experiments in this particular application of digital signal processing.

I strongly believe that security-by-obscurity is the wrong way to offer any kind of forensic service (i.e. "Using this proprietary software I guarantee you that this photo is pristine... and you have to trust me!"). Following the open-source philosophy, everyone should be able to try various techniques on their own, gain knowledge and share it to the community... even better if they contribute with code improvements! :)

• History
• Features
• Screenshots
• Installation
• Updates
• Bibliography

History

The first version was written in 2015 using C++11 to build a command line utility with many options, but soon it turned to be too cumbersome and not much interactive. That version could be compiled with CMake after installing OpenCV, Boost and AlgLib libraries. This first proof of concept offered about 80% of planned features (see below for the full list).

While also including novel algorithms, the 2017 version mainly added a Qt-based multi-window GUI to provide a better user experience. Multiple analyses could be shown on screen and a fast zoom/scroll viewer was implemented for easier image navigation. That project could be compiled with Qt Creator with Qt 5 and OpenCV 3 and covered about 70% of planned features.

Fast-forward to 2020 when I decided to port everything in Python (PySide2 + Matplotlib + OpenCV) for easier development and deployment. While this iteration is just begun and I have yet to port all the previous code on the new platform, I think this will be the final "form" of the project (as long as someone does not volunteer up to develop a nice web application!).

I'm happy to share my code and get in contact with anyone interested to improve or test it, but please keep in mind that this repository is not intended for distributing a final product, my aim is just to publicly track development of an unpretentious educational tool, so expect bugs, unpolished code and missing features! ;)

Features

This list contains the functions that the toolkit will (hopefully) provide once beta stage is reached (NOTE: functions displayed in italics inside the program are not yet implemented!).

Interface

• Modern Qt-based GUI with multiple tool window management
• Support for many formats (JPEG, PNG, TIFF, BMP, WebP, PGM, PFM, GIF)
• Highly responsive image viewer with real-time pan and zoom
• Many state-of-the-art algorithms to try out interactively
• Export both visual and textual results of the analysis
• Extensive online help with explanations and tutorials

Tools

General

• Original Image: display the unaltered reference image for visual inspection
• File Digest: retrieve physical file information, crypto and perceptual hashes
• Hex Editor: open an external hexadecimal editor to show and edit raw bytes
• Similar Search: browse online search services to find visually similar images

Metadata

• Header Structure: dump the file header structure and display an interactive view
• EXIF Full Dump: scan through file metadata and gather all available information
• Thumbnail Analysis: extract optional embedded thumbnail and compare with original
• Geolocation Data: retrieve optional geolocation data and show it on a world map

Inspection

• Enhancing Magnifier: magnifying glass with enhancements for better identifying forgeries
• Channel Histogram: display single color channels or RGB composite interactive histogram
• Global Adjustments: apply standard image adjustments (brightness, hue, saturation, ...)
• Reference Comparison: open a synchronized double view for comparison with another picture

Detail

• Luminance Gradient: analyze horizontal/vertical brightness variations across the image
• Echo Edge Filter: use derivative filters to reveal artificial out-of-focus regions
• Wavelet Threshold: reconstruct image with different wavelet coefficient thresholds
• Frequency Split: split image luminance into high and low frequency components

Colors

• RGB/HSV Plots: display interactive 2D and 3D plots of RGB and HSV pixel values
• Space Conversion: convert RGB channels into HSV/YCbCr/Lab/Luv/CMYK/Gray spaces
• PCA Projection: use color PCA to project pixel onto most salient components
• Pixel Statistics: compute minimum/maximum/average RGB values for every pixel

Noise

• Noise Separation: estimate and extract different kind of image noise components
• Min/Max Deviation: highlight pixels deviating from block-based min/max statistics
• Bit Planes Values: show individual bit planes to find inconsistent noise patterns
• PRNU Identification: exploit sensor pattern noise introduced by different cameras

JPEG

• Quality Estimation: extract quantization tables and estimate last saved JPEG quality
• Error Level Analysis: show pixel-level difference against fixed compression levels
• Multiple Compression: use a machine learning model to detect multiple compression
• JPEG Ghost Maps: highlight traces of different compression levels in difference images

Tampering

• Contrast Enhancement: analyze color distribution to detect contrast enhancements
• Copy-Move Forgery: use invariant feature descriptors for cloned area detection
• Composite Splicing: exploit DCT statistics for automatic splicing zone detection
• Image Resampling: estimate 2D pixel interpolation for detecting resampling traces

Various

• Median Filtering: detect processing traces left by nonlinear median filtering
• Illuminant Map: estimate scene local light direction on estimated 3D surfaces
• Dead/Hot Pixels: detect and fix dead/hot pixels caused by sensor imperfections
• Stereogram Decoder: decode 3D images concealed in crossed-eye autostereograms

Screenshots

General: Original Image, Hex Editor, File Digest, Similar Search

Metadata: EXIF Full Dump, Header Structure

Inspection: Enhancing Magnifier, Channel Histogram, Reference Comparison

Detail: Luminance Gradient, Echo Edge Filter, Wavelet Threshold, Frequency Split

Colors: RGB/HSV Plots, Space Conversion, PCA Projection, Pixel Statistics

Noise: Signal Separation, Min/Max Deviation, Bit Plane Values

JPEG: Quality Estimation, Error Level Analysis

Tampering: Contrast Enhancement, Copy/Move Forgery, Composite Splicing, Median Filtering

Installation

[1/4] Donwload source code

Clone the current repository into a local folder and change current directory to it.

[2/4] Virtual environment

For more information about Python Virtual Environments, you can read here or here. Choose one of the following method to create a new virtual environment with Python 3.11 for Sherloq.

Built-in Virtual Environment

Change current directory to Sherloq root, then initialize virtual environment folder:

$ python -m venv .venv

Then activate it:

Linux

$ source .venv/bin/activate

Windows

C:\> .venv\Scripts\activate.bat

VirtualEnvWrapper

Linux

$ sudo apt install python3-distutils python3-dev python3-testresources subversion
$ wget https://bootstrap.pypa.io/get-pip.py
$ sudo python3 get-pip.py
$ rm get-pip.py
$ sudo pip install virtualenv virtualenvwrapper
$ echo -e "\n# Python Virtual Environments" >> ~/.bashrc
$ echo "export WORKON_HOME=$HOME/.virtualenvs" >> ~/.bashrc
$ echo "export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3" >> ~/.bashrc
$ echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc
$ source ~/.bashrc
$ mkvirtualenv sq -p python3

Windows

1. Download Python 3.11 setup package from official site
2. Install ensuring that "Add Python to PATH" and "PIP installation" are enabled
3. Open Command Prompt and enter the following commands:

> pip install virtualenv virtualenvwrapper-win
> mkvirtualenv sq

Conda environment

1. Install latest Miniconda following instructions from the official site
2. Create a new virtual environment: conda create --name sq python=3.11 -y
3. Activate the sq environment: conda activate sq

[3/4] Install dependencies

pip install -r requirements.txt

[4/4] Launch program

python sherloq.py

Updates

When a new version is released, update the local working copy using Git, SVN or manually downloading from this repository and (if necessary) update the packages in the virtual environment following this guide.

Bibliography

• "A Picture's Worth: Digital Image Analysis and Forensics" (Neal Krawetz) [paper]
• "Noiseprint: a CNN-based camera model fingerprint" (Davide Cozzolino, Luisa Verdoliva) [website]
• "Exposing Digital Forgeries by Detecting Traces of Re-sampling" (Alin C. Popescu and Hany Farid) [paper]
• "Two Improved Forensic Methods of Detecting Contrast Enhancement in Digital Images" (Xufeng Lin, Xingjie Wei and Chang-Tsun Li) [paper]

sherloq's Issues

MacOS installation problem

there seems to be another conflict that I could not solve

(sherloq) (base) user@vivi13 gui % python sherloq.py                 
objc[17106]: Class QMacAutoReleasePoolTracker is implemented in both /Users/user/.virtualenvs/sherloq/lib/python3.7/site-packages/PySide2/Qt/lib/QtCore.framework/Versions/5/QtCore (0x10a4fe0f8) and /Users/user/.virtualenvs/sherloq/lib/python3.7/site-packages/cv2/.dylibs/QtCore (0x11d843700). One of the two will be used. Which one is undefined.
objc[17106]: Class QT_ROOT_LEVEL_POOL__THESE_OBJECTS_WILL_BE_RELEASED_WHEN_QAPP_GOES_OUT_OF_SCOPE is implemented in both /Users/user/.virtualenvs/sherloq/lib/python3.7/site-packages/PySide2/Qt/lib/QtCore.framework/Versions/5/QtCore (0x10a4fe170) and /Users/user/.virtualenvs/sherloq/lib/python3.7/site-packages/cv2/.dylibs/QtCore (0x11d843778). One of the two will be used. Which one is undefined.
objc[17106]: Class KeyValueObserver is implemented in both /Users/user/.virtualenvs/sherloq/lib/python3.7/site-packages/PySide2/Qt/lib/QtCore.framework/Versions/5/QtCore (0x10a4fe198) and /Users/user/.virtualenvs/sherloq/lib/python3.7/site-packages/cv2/.dylibs/QtCore (0x11d8437a0). One of the two will be used. Which one is undefined.
objc[17106]: Class RunLoopModeTracker is implemented in both /Users/user/.virtualenvs/sherloq/lib/python3.7/site-packages/PySide2/Qt/lib/QtCore.framework/Versions/5/QtCore (0x10a4fe1e8) and /Users/user/.virtualenvs/sherloq/lib/python3.7/site-packages/cv2/.dylibs/QtCore (0x11d8437f0). One of the two will be used. Which one is undefined.
objc[17106]: Class QCocoaPageLayoutDelegate is implemented in both /Users/user/.virtualenvs/sherloq/lib/python3.7/site-packages/PySide2/Qt/lib/QtPrintSupport.framework/Versions/5/QtPrintSupport (0x12c78d540) and /Users/user/.virtualenvs/sherloq/lib/python3.7/site-packages/cv2/.dylibs/QtPrintSupport (0x151ef3468). One of the two will be used. Which one is undefined.
objc[17106]: Class QCocoaPrintPanelDelegate is implemented in both /Users/user/.virtualenvs/sherloq/lib/python3.7/site-packages/PySide2/Qt/lib/QtPrintSupport.framework/Versions/5/QtPrintSupport (0x12c78d5b8) and /Users/user/.virtualenvs/sherloq/lib/python3.7/site-packages/cv2/.dylibs/QtPrintSupport (0x151ef34e0). One of the two will be used. Which one is undefined.
QObject::moveToThread: Current thread (0x7fd975ed4df0) is not the object's thread (0x7fd976b52c70).
Cannot move to target thread (0x7fd975ed4df0)
You might be loading two sets of Qt binaries into the same process. Check that all plugins are compiled against the right Qt binaries. Export DYLD_PRINT_LIBRARIES=1 and check that only one set of binaries are being loaded.
qt.qpa.plugin: Could not load the Qt platform plugin "cocoa" in "/Users/user/.virtualenvs/sherloq/lib/python3.7/site-packages/cv2/qt/plugins" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: cocoa, minimal, offscreen, webgl.

zsh: abort      python sherloq.py

Originally posted by @under-score in #5 (comment)

Load image dialogue - macOS

On macOS all views in the Load image dialogue show icons like "JPEG", "PNG" etc. - is this platform specific and is there a way to make it show thumbnail previews as the system dialogue would?

Windows compatibility

When subprocess module is called under Windows, the program remains stuck on the prompt waiting for ENTER key to be pressed (see here for a possible solution).

installation

downloaded the file. unable to install. using kali 2020.2. need instructions. do i have to compile first?

Error: " *args, **kwargs "

When running sherloq.py, the following message appears:

Traceback (most recent call last):
File "E:\GitHub\sherloq-master\gui\sherloq.py", line 20, in
from histogram import HistWidget
File "C:\Users\Griffon\AppData\Local\Programs\Python\Python39\Lib\site-packages\shiboken2\files.dir\shibokensupport_feature_.py", line 142, in import
return original_import(name, *args, **kwargs)
File "E:\GitHub\sherloq-master\gui\histogram.py", line 17, in
from matplotlib.backends.backend_qt5agg import FigureCanvas
File "C:\Users\Griffon\AppData\Local\Programs\Python\Python39\Lib\site-packages\shiboken2\files.dir\shibokensupport_feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
ModuleNotFoundError: No module named 'matplotlib'

Web Interface

My experience is more with PHP/JS but I am not convinced to marry both worlds by some
shell_exec('python hello.py');
Isn't there any Django / Turbogears programmer out there to jump in?

lib not found while final installation

SyntaxError: invalid syntax

I got this error from python sherloq.py
Have no error while installing

xxx@xxx:~/sherloq/sherloq/gui$ python sherloq.py
File "sherloq.py", line 53
self.setWindowTitle(f"{QApplication.applicationName()} {QApplication.applicationVersion()}")
^
SyntaxError: invalid syntax

Any advice, need pre-installed libraries or something more to do?
running
Python 2.7.18
Python 3.8.10

load_image

Traceback (most recent call last):
File "sherloq.py", line 218, in load_file
self.filename, basename, self.image = load_image(self)
ValueError: not enough values to unpack (expected 3, got 2)

ImportError('failed to find libmagic. Check your installation') on windows

after following the installation procedure for windows (using Python 3.9.2)

(sq) D:\Download\sherloq-master\sherloq-master\gui>pip install -r requirements_win.txt
Collecting tensorflow
Downloading tensorflow-2.6.0-cp39-cp39-win_amd64.whl (423.3 MB)
|████████████████████████████████| 423.3 MB 48 kB/s
Collecting astor
Downloading astor-0.8.1-py2.py3-none-any.whl (27 kB)
Collecting concurrent-iterator
Downloading concurrent_iterator-0.2.6-py2.py3-none-any.whl (7.5 kB)
Collecting keras-applications
Downloading Keras_Applications-1.0.8-py3-none-any.whl (50 kB)
|████████████████████████████████| 50 kB 1.6 MB/s
Collecting lxml
Downloading lxml-4.6.3-cp39-cp39-win_amd64.whl (3.5 MB)
|████████████████████████████████| 3.5 MB 439 kB/s
Collecting opencv-contrib-python-headless
Downloading opencv_contrib_python_headless-4.5.3.56-cp39-cp39-win_amd64.whl (41.7 MB)
|████████████████████████████████| 41.7 MB 1.7 MB/s
Collecting pandas
Downloading pandas-1.3.3-cp39-cp39-win_amd64.whl (10.2 MB)
|████████████████████████████████| 10.2 MB 1.1 MB/s
Collecting pyside2
Downloading PySide2-5.15.2-5.15.2-cp35.cp36.cp37.cp38.cp39-none-win_amd64.whl (136.3 MB)
|████████████████████████████████| 136.3 MB 62 kB/s
Collecting python-magic
Downloading python_magic-0.4.24-py2.py3-none-any.whl (12 kB)
Collecting python-magic-bin
Downloading python_magic_bin-0.4.14-py2.py3-none-win_amd64.whl (409 kB)
|████████████████████████████████| 409 kB 1.6 MB/s
Collecting rawpy
Downloading rawpy-0.16.0-cp39-cp39-win_amd64.whl (643 kB)
|████████████████████████████████| 643 kB 1.7 MB/s
Collecting scikit-image
Downloading scikit_image-0.18.3-cp39-cp39-win_amd64.whl (12.2 MB)
|████████████████████████████████| 12.2 MB 1.6 MB/s
Collecting scikit-learn
Downloading scikit_learn-0.24.2-cp39-cp39-win_amd64.whl (6.9 MB)
|████████████████████████████████| 6.9 MB 819 kB/s
Collecting sewar
Downloading sewar-0.4.4.tar.gz (10 kB)
Collecting xgboost==1.1.1
Downloading xgboost-1.1.1-py3-none-win_amd64.whl (54.4 MB)
|████████████████████████████████| 54.4 MB 1.6 MB/s
Collecting numpy
Downloading numpy-1.21.2-cp39-cp39-win_amd64.whl (14.0 MB)
|████████████████████████████████| 14.0 MB 1.7 MB/s
Collecting scipy
Downloading scipy-1.7.1-cp39-cp39-win_amd64.whl (33.8 MB)
|████████████████████████████████| 33.8 MB 208 kB/s
Collecting tensorflow-estimator~=2.6
Downloading tensorflow_estimator-2.6.0-py2.py3-none-any.whl (462 kB)
|████████████████████████████████| 462 kB 1.7 MB/s
Collecting protobuf>=3.9.2
Downloading protobuf-3.18.0-cp39-cp39-win_amd64.whl (912 kB)
|████████████████████████████████| 912 kB 1.7 MB/s
Collecting numpy
Downloading numpy-1.19.5-cp39-cp39-win_amd64.whl (13.3 MB)
|████████████████████████████████| 13.3 MB 1.6 MB/s
Collecting wrapt~=1.12.1
Downloading wrapt-1.12.1.tar.gz (27 kB)
Collecting keras-preprocessing~=1.1.2
Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
|████████████████████████████████| 42 kB 462 kB/s
Collecting keras~=2.6
Downloading keras-2.6.0-py2.py3-none-any.whl (1.3 MB)
|████████████████████████████████| 1.3 MB 1.6 MB/s
Collecting gast==0.4.0
Downloading gast-0.4.0-py3-none-any.whl (9.8 kB)
Collecting tensorboard~=2.6
Downloading tensorboard-2.6.0-py3-none-any.whl (5.6 MB)
|████████████████████████████████| 5.6 MB 1.6 MB/s
Requirement already satisfied: wheel~=0.35 in c:\users\wfrey\envs\sq\lib\site-packages (from tensorflow->-r requirements_win.txt (line 1)) (0.37.0)
Collecting termcolor~=1.1.0
Downloading termcolor-1.1.0.tar.gz (3.9 kB)
Collecting opt-einsum~=3.3.0
Downloading opt_einsum-3.3.0-py3-none-any.whl (65 kB)
|████████████████████████████████| 65 kB 1.0 MB/s
Collecting six~=1.15.0
Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting flatbuffers~=1.12.0
Downloading flatbuffers-1.12-py2.py3-none-any.whl (15 kB)
Collecting clang~=5.0
Downloading clang-5.0.tar.gz (30 kB)
Collecting grpcio<2.0,>=1.37.0
Downloading grpcio-1.40.0-cp39-cp39-win_amd64.whl (3.2 MB)
|████████████████████████████████| 3.2 MB 1.6 MB/s
Collecting typing-extensions~=3.7.4
Downloading typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting astunparse~=1.6.3
Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting google-pasta~=0.2
Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
|████████████████████████████████| 57 kB 1.1 MB/s
Collecting h5py~=3.1.0
Downloading h5py-3.1.0-cp39-cp39-win_amd64.whl (2.7 MB)
|████████████████████████████████| 2.7 MB 1.3 MB/s
Collecting absl-py~=0.10
Downloading absl_py-0.13.0-py3-none-any.whl (132 kB)
|████████████████████████████████| 132 kB 1.3 MB/s
Collecting decorator
Downloading decorator-5.1.0-py3-none-any.whl (9.1 kB)
Collecting pytz>=2017.3
Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB)
|████████████████████████████████| 510 kB 1.7 MB/s
Collecting python-dateutil>=2.7.3
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
|████████████████████████████████| 247 kB 3.3 MB/s
Collecting shiboken2==5.15.2
Downloading shiboken2-5.15.2-5.15.2-cp35.cp36.cp37.cp38.cp39-none-win_amd64.whl (2.3 MB)
|████████████████████████████████| 2.3 MB 1.6 MB/s
Collecting networkx>=2.0
Downloading networkx-2.6.3-py3-none-any.whl (1.9 MB)
|████████████████████████████████| 1.9 MB 1.6 MB/s
Collecting tifffile>=2019.7.26
Downloading tifffile-2021.8.30-py3-none-any.whl (171 kB)
|████████████████████████████████| 171 kB 1.6 MB/s
Collecting imageio>=2.3.0
Downloading imageio-2.9.0-py3-none-any.whl (3.3 MB)
|████████████████████████████████| 3.3 MB 6.8 MB/s
Collecting PyWavelets>=1.1.1
Downloading PyWavelets-1.1.1-cp39-cp39-win_amd64.whl (4.2 MB)
|████████████████████████████████| 4.2 MB 1.7 MB/s
Collecting pillow!=7.1.0,!=7.1.1,>=4.3.0
Downloading Pillow-8.3.2-cp39-cp39-win_amd64.whl (3.2 MB)
|████████████████████████████████| 3.2 MB 1.6 MB/s
Collecting matplotlib!=3.0.0,>=2.0.0
Downloading matplotlib-3.4.3-cp39-cp39-win_amd64.whl (7.1 MB)
|████████████████████████████████| 7.1 MB 1.7 MB/s
Collecting threadpoolctl>=2.0.0
Downloading threadpoolctl-2.2.0-py3-none-any.whl (12 kB)
Collecting joblib>=0.11
Downloading joblib-1.0.1-py3-none-any.whl (303 kB)
|████████████████████████████████| 303 kB 1.6 MB/s
Collecting pyparsing>=2.2.1
Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
|████████████████████████████████| 67 kB 1.3 MB/s
Collecting cycler>=0.10
Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting kiwisolver>=1.0.1
Downloading kiwisolver-1.3.2-cp39-cp39-win_amd64.whl (52 kB)
|████████████████████████████████| 52 kB 331 kB/s
Requirement already satisfied: setuptools>=41.0.0 in c:\users\wfrey\envs\sq\lib\site-packages (from tensorboard~=2.6->tensorflow->-r requirements_win.txt (line 1)) (58.0.4)
Collecting werkzeug>=0.11.15
Downloading Werkzeug-2.0.1-py3-none-any.whl (288 kB)
|████████████████████████████████| 288 kB 1.7 MB/s
Collecting markdown>=2.6.8
Downloading Markdown-3.3.4-py3-none-any.whl (97 kB)
|████████████████████████████████| 97 kB 1.4 MB/s
Collecting google-auth-oauthlib<0.5,>=0.4.1
Downloading google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Collecting requests<3,>=2.21.0
Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
|████████████████████████████████| 62 kB 2.3 MB/s
Collecting tensorboard-plugin-wit>=1.6.0
Downloading tensorboard_plugin_wit-1.8.0-py3-none-any.whl (781 kB)
|████████████████████████████████| 781 kB 2.2 MB/s
Collecting tensorboard-data-server<0.7.0,>=0.6.0
Downloading tensorboard_data_server-0.6.1-py3-none-any.whl (2.4 kB)
Collecting google-auth<2,>=1.6.3
Downloading google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
|████████████████████████████████| 152 kB 1.6 MB/s
Collecting pyasn1-modules>=0.2.1
Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
|████████████████████████████████| 155 kB 1.6 MB/s
Collecting rsa<5,>=3.1.4
Downloading rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting cachetools<5.0,>=2.0.0
Downloading cachetools-4.2.2-py3-none-any.whl (11 kB)
Collecting requests-oauthlib>=0.7.0
Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting pyasn1<0.5.0,>=0.4.6
Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
|████████████████████████████████| 77 kB 1.7 MB/s
Collecting certifi>=2017.4.17
Downloading certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
|████████████████████████████████| 145 kB 1.6 MB/s
Collecting idna<4,>=2.5
Downloading idna-3.2-py3-none-any.whl (59 kB)
|████████████████████████████████| 59 kB 1.2 MB/s
Collecting charset-normalizer~=2.0.0
Downloading charset_normalizer-2.0.5-py3-none-any.whl (37 kB)
Collecting urllib3<1.27,>=1.21.1
Downloading urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
|████████████████████████████████| 138 kB 1.7 MB/s
Collecting oauthlib>=3.0.0
Downloading oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
|████████████████████████████████| 146 kB 1.7 MB/s
Building wheels for collected packages: sewar, clang, termcolor, wrapt
Building wheel for sewar (setup.py) ... done
Created wheel for sewar: filename=sewar-0.4.4-py3-none-any.whl size=10390 sha256=2b59b8530d8795c03334c4bb8f0c7d332fc23b46b6b902247f91606bbe8e092d
Stored in directory: c:\users\wfrey\appdata\local\pip\cache\wheels\49\a0\32\0abb7141de45c9d76e08efaffa755a9939c093251231b90259
Building wheel for clang (setup.py) ... done
Created wheel for clang: filename=clang-5.0-py3-none-any.whl size=30692 sha256=584e0cb609ba76317a578516602b5708654ec580706132a6b3c1cbd3b9f9acab
Stored in directory: c:\users\wfrey\appdata\local\pip\cache\wheels\3a\ce\7a\27094f689461801c934296d07078773603663dfcaca63bb064
Building wheel for termcolor (setup.py) ... done
Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4847 sha256=6d70dd14df453f620491bccfcbf32eac23b51572c76075df5f17521c328ddee8
Stored in directory: c:\users\wfrey\appdata\local\pip\cache\wheels\b6\0d\90\0d1bbd99855f99cb2f6c2e5ff96f8023fad8ec367695f7d72d
Building wheel for wrapt (setup.py) ... done
Created wheel for wrapt: filename=wrapt-1.12.1-py3-none-any.whl size=19572 sha256=736cebd57ab8aaa0ef5d7039c8ae8a0f598c88554bf6f88c4d0dd59b5eccb9ac
Stored in directory: c:\users\wfrey\appdata\local\pip\cache\wheels\98\23\68\efe259aaca055e93b08e74fbe512819c69a2155c11ba3c0f10
Successfully built sewar clang termcolor wrapt
Installing collected packages: urllib3, pyasn1, idna, charset-normalizer, certifi, six, rsa, requests, pyasn1-modules, oauthlib, cachetools, requests-oauthlib, google-auth, werkzeug, tensorboard-plugin-wit, tensorboard-data-server, python-dateutil, pyparsing, protobuf, pillow, numpy, markdown, kiwisolver, grpcio, google-auth-oauthlib, cycler, absl-py, wrapt, typing-extensions, tifffile, threadpoolctl, termcolor, tensorflow-estimator, tensorboard, shiboken2, scipy, PyWavelets, pytz, opt-einsum, networkx, matplotlib, keras-preprocessing, keras, joblib, imageio, h5py, google-pasta, gast, flatbuffers, decorator, clang, astunparse, xgboost, tensorflow, sewar, scikit-learn, scikit-image, rawpy, python-magic-bin, python-magic, pyside2, pandas, opencv-contrib-python-headless, lxml, keras-applications, concurrent-iterator, astor
Successfully installed PyWavelets-1.1.1 absl-py-0.13.0 astor-0.8.1 astunparse-1.6.3 cachetools-4.2.2 certifi-2021.5.30 charset-normalizer-2.0.5 clang-5.0 concurrent-iterator-0.2.6 cycler-0.10.0 decorator-5.1.0 flatbuffers-1.12 gast-0.4.0 google-auth-1.35.0 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 grpcio-1.40.0 h5py-3.1.0 idna-3.2 imageio-2.9.0 joblib-1.0.1 keras-2.6.0 keras-applications-1.0.8 keras-preprocessing-1.1.2 kiwisolver-1.3.2 lxml-4.6.3 markdown-3.3.4 matplotlib-3.4.3 networkx-2.6.3 numpy-1.19.5 oauthlib-3.1.1 opencv-contrib-python-headless-4.5.3.56 opt-einsum-3.3.0 pandas-1.3.3 pillow-8.3.2 protobuf-3.18.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pyparsing-2.4.7 pyside2-5.15.2 python-dateutil-2.8.2 python-magic-0.4.24 python-magic-bin-0.4.14 pytz-2021.1 rawpy-0.16.0 requests-2.26.0 requests-oauthlib-1.3.0 rsa-4.7.2 scikit-image-0.18.3 scikit-learn-0.24.2 scipy-1.7.1 sewar-0.4.4 shiboken2-5.15.2 six-1.15.0 tensorboard-2.6.0 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.0 tensorflow-2.6.0 tensorflow-estimator-2.6.0 termcolor-1.1.0 threadpoolctl-2.2.0 tifffile-2021.8.30 typing-extensions-3.7.4.3 urllib3-1.26.6 werkzeug-2.0.1 wrapt-1.12.1 xgboost-1.1.1

I get the following error:

(sq) D:\Download\sherloq-master\sherloq-master\gui>python sherloq.py
Traceback (most recent call last):
File "D:\Download\sherloq-master\sherloq-master\gui\sherloq.py", line 12, in
from digest import DigestWidget
File "C:\Users\wfrey\Envs\sq\Lib\site-packages\shiboken2\files.dir\shibokensupport_feature_.py", line 142, in import
return original_import(name, *args, **kwargs)
File "D:\Download\sherloq-master\sherloq-master\gui\digest.py", line 5, in
import magic
File "C:\Users\wfrey\Envs\sq\Lib\site-packages\shiboken2\files.dir\shibokensupport_feature.py", line 142, in import
return original_import(name, *args, **kwargs)
File "C:\Users\wfrey\Envs\sq\lib\site-packages\magic_init.py", line 208, in
libmagic = loader.load_lib()
File "C:\Users\wfrey\Envs\sq\lib\site-packages\magic\loader.py", line 49, in load_lib
raise ImportError('failed to find libmagic. Check your installation')
ImportError: failed to find libmagic. Check your installation

(sq) D:\Download\sherloq-master\sherloq-master\gui>

Run python sherloq.py get TypeError

python sherloq.py
Traceback (most recent call last):
File "/home/alex/PycharmProjects/sherloq/gui/sherloq.py", line 23, in
from median import MedianWidget
File "/usr/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/home/alex/PycharmProjects/sherloq/gui/median.py", line 15, in
from joblib import load
File "/usr/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/joblib/init.py", line 119, in
from .parallel import Parallel
File "/usr/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/joblib/parallel.py", line 28, in
from ._parallel_backends import (FallbackToBackend, MultiprocessingBackend,
File "/usr/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/joblib/_parallel_backends.py", line 22, in
from .executor import get_memmapping_executor
File "/usr/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/joblib/executor.py", line 14, in
from .externals.loky.reusable_executor import get_reusable_executor
File "/usr/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/joblib/externals/loky/init.py", line 12, in
from .backend.reduction import set_loky_pickler
File "/usr/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/joblib/externals/loky/backend/reduction.py", line 125, in
from joblib.externals import cloudpickle # noqa: F401
File "/usr/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/joblib/externals/cloudpickle/init.py", line 3, in
from .cloudpickle import *
File "/usr/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 152, in
_cell_set_template_code = _make_cell_set_template_code()
File "/usr/lib/python3.9/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 133, in _make_cell_set_template_code
return types.CodeType(
TypeError: an integer is required (got type bytes)

Header Structure not working as intended (invalid URL for file)

Hey,

if I try to open the "Header Structure", the WebView does not open the file properly. It tries to open file://c/Users/Lorenz/AppData/Local/Temp/Sherloq-mcAXVc%5Cstructure.html when instead file:///C:/Users/Lorenz/AppData/Local/Temp/Sherloq-mcAXVc/structure.html should be opened.

I'm running the most recent version of sherloq on a Windows Machine.

Is this work done? I don't find dct dimples map...

Hi,
thanks for your work！！
I wonder if the dct dimples map has been done? Cause I don't find it .....

Is there any scientific guide to tools?

I am very amazed with all the tools provided for image forensics in this repo which certainly got a lot of time to research.

I opened this issue because I cant find good resource that can provide full description and mathematical theories behind all the tools that have been used here like error level analysis or luminance gradient and etc.

please provide some information and mathematical description behind the developed tools and include it in the README.md file.

Thanks

Problema de Instalação no WSL do Windows

Fui tentar instalar na WSL do Windows no Ubuntu 20.04 LTS mais não foi possível.

WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
ERROR: Could not find a version that satisfies the requirement pip (from versions: none)
ERROR: No matching distribution found for pip
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping

andre@DESKTOP-8SPDF08:~/sherloq/gui$ python sherloq.py

Traceback (most recent call last):
File "/home/andre/sherloq/gui/sherloq.py", line 4, in
from PySide2.QtCore import Qt, QSettings
ModuleNotFoundError: No module named 'PySide2'
andre@DESKTOP-8SPDF08:~/sherloq/gui$

stress test cloning.py

when testing now a large number of files on the command line, I could identify a bmp and a jpg file that run forever
Archiv.zip

UX issue

In Contrast Enhancement, Process runs for all algorithms then the user selects their favourite to display. But in others e.g. Copy-Move Forgery the parameters are set before the Process button is hit. Could these steps easily be made more consistent?

Export capability - proposal & motivation

This builds on #28 but is a wider feature request that I would like to motivate for here. @GuidoBartoli Please shout where you agree and disagree below.

Sherloq provides excellent tools for visual analysis and it does this job well.

Numerical analysis is an important component of forensic analysis. This could be used, for example, to build a tampering detector that runs blind without human intervention. This could be incorporated but may be out of scope for Sherloq's intended purpose. Broadcasting to another python instance could also be helpful but probably hard to implement.

There are many good tools for such analysis. One example from my previous field of astronomy(*) is saoimage ds9 (https://sites.google.com/cfa.harvard.edu/saoimageds9). It is well established and contains a rich feature set (too many to mention but:) including,

• Pan Zoom Rotate
• Alignment of tiles
• Overlaying of regions
• Import/export
• Python integration/broadcasting
• Contouring
• A full suite of colo(u)r schemes / rescaling / equalization
• Continually updated box containing coordinates, values, ...
• Histogram cuts
• Support for a range of image formats (but mainly astronomy focused)

(*) Probably every field has its favourite tools, however; I would be loathe to lock Sherloq into any particular field's toolset, awesome as ds9 is.

I imagine some of these would be either (i) straightforward or (ii) useful to add to Sherloq. But for those that are neither why reinvent the wheel?

A quick-kill route to accessing the above functionality - in addition to the existing export capability(+) - would be a comprehensive export button geared towards numerical analysis and development of blind, automated pipelines. This could simply be export to .npy (for all images) or .pkl for other variables and would afford flexibility by avoiding field lock-in. Examples of other possible variables might be

• Algorithm settings (which would allow for reproducibility - since these could then in principle be loaded back in for reruns)
• Keypoints (these can in fact be serialized into .pkl - I have functions for this; also for reload)
• Contour sets (format?)
• EXIF (to e.g. CSV/JSON)
• Thumbnails
• Sherloq hallmark to ensure and validate integrity of pipeline by, for example, commit versioning (though could this be hacked/spoofed?)

(+) I take the point about sticking to PNG for export because it is lossless. Of course JPEG compression is lossy and a critical consequence of this - I think - is (repeatable) quantization noise. But unless the exported images are being used for downstream numerical analysis (see above) then does it matter? The original image is - of course and importantly - not being modified. Having the output as PNG when the input is JPEG - or vice versa - in fact I think hinders further analysis/comparison. For example, what tool/s would allow easy overlay of such images - GIMP (to be discouraged I imagine!)..?

@GuidoBartoli Let me know your thoughts - I'd be happy to help work on the above if you think any of it would be useful (see also #29).

And just to reiterate that Sherloq is unique, excellent and of great value!

[Conda] version `Qt_5.15' not found (required by PySide2)

Hi

I am trying to install sherloq through conda +pip (on CentOS-7) but it seems that the PySide2 is using a newer qt5 version than the conda provided one:

(sherloq-pip) [tru@elitebook840g3 gui]$ python3 sherloq.py 
Traceback (most recent call last):
  File "sherloq.py", line 4, in <module>
    from PySide2.QtCore import Qt, QSettings
ImportError: /c7/shared/anaconda3/lib/libQt5Core.so.5: version `Qt_5.15' not found (required by /c7/shared/anaconda3/envs/sherloq-pip/lib/python3.8/site-packages/PySide2/QtCore.abi3.so)

Procedure used:

• install anaconda3
• conda update anaconda && conda update --all
• conda create --copy -n sherloq-pip python
• conda activate sherloq-pip
• git clone https://github.com/GuidoBartoli/sherloq
• cd sherloq/gui
• python3 -m pip install -r requirements.txt
• python3 sherloq.py

Issue with Splicing Probability Heatmap

In the image below, there are some red circular artifacts. What is the significance of these? Is there a way to mitigate them to see what the underlying heatmap looks like?

Thanks!

BRISK versus NoisePrint

NoisePrint is a great new feature.

Here is a quick photoshopped file (content aware fill of the stone bottom right)

that is perfectly recognized by BRISK while the noise heatmap is quite off target...

some tools are in italics and don't work

list:
Illuminant Map
PRNU Identification
Multiple Compression
JPEG Ghost Maps
Image Resampling
Dead\Hot Pixels

doesn't start anymore

even after updating requirements.txt with tensorflow=1.15 and brew install libmagic the macOS Big Sur terminal hangs forever showing the warning

WARNING:tensorflow:From tilde/.virtualenvs/Sherloq/lib/python3.7/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating: non-resource variables are not supported in the long term

Linux [ubuntu] install issues

Hi!

I am trying to install on Ubuntu 16.04 your library.

I met the following error while installing requirements.txt:

ERROR: tensorflow 2.2.0 has requirement scipy==1.4.1; python_version >= "3", but you'll have scipy 1.5.1 which is incompatible.

Ok, ignored, moving forward.

When launching, I get:

python sherloq.py
WARNING:tensorflow:From /home/*****/miniconda3/envs/serloq/lib/python3.8/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, webgl, xcb.

[1]    5775 abort (core dumped)  python sherloq.py

[I am using anaconda instead of virtualenv, but the starting env was empty (eg python 3.8 only) ]

Tried to fiddle with opencv (headless, contrib) and pyqt5. The 2nd problem still persists.

LE: I ran sudo apt-get install libxcb-xinerama0 as found here: https://forum.qt.io/topic/93247/qt-qpa-plugin-could-not-load-the-qt-platform-plugin-xcb-in-even-though-it-was-found

source ~/.bash_profile

i am on Mac, I followed the installation instruction, and when I type source ~/.bash_profile, I get this :

-sh: -e: command not found
-sh: /usr/local/bin/virtualenvwrapper.sh: No such file or directory

help please
thanks

PYSIDE2 error

Traceback (most recent call last): File "/data/data/com.termux/files/home/sherloq/gui/sherloq.py", line 4, in <module> from PySide2.QtCore import Qt, QSettings ModuleNotFoundError: No module named 'PySide2'

missing dependency `matplotlib`

Hi,

I suppose that requirements.txt might be missing a matplotlib entry :

After installing sherloq in a virtual Python environment according to the instructions, starting sherloq fails, yielding the following exception :

( sq)[ master|/dev/shm/sherloq/gui/]
$ python sherloq.py
Traceback (most recent call last):
  File "/dev/shm/sherloq/gui/sherloq.py", line 20, in <module>
    from histogram import HistWidget
  File "/dev/shm/.venv/sq/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/dev/shm/sherloq/gui/histogram.py", line 17, in <module>
    from matplotlib.backends.backend_qt5agg import FigureCanvas
  File "/dev/shm/.venv/sq/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
ModuleNotFoundError: No module named 'matplotlib'

After

( sq)[ master|/dev/shm/sherloq/gui/]
$ pip install matplotlib

inside the venv, I can successfully start Sherloq.

---

Environment :

OS	python	virtualenv	virtualenvwrapper	shiboken2	sherloq
Archlinux	3.9.9	20.8.0	4.8.4	5.15.2	[master]

source ~/.bash_profile

hi

this is my problem while entering the last command:
source ~/.bash_profile

`source ~/.bash_profile
/Library/Developer/CommandLineTools/usr/bin/python3: Error while finding module specification for 'virtualenvwrapper.hook_loader' (ModuleNotFoundError: No module named 'virtualenvwrapper')
virtualenvwrapper.sh: There was a problem running the initialization hooks.

If Python could not import the module virtualenvwrapper.hook_loader,
check that virtualenvwrapper has been installed for
VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3 and that PATH is
set properly.
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: No module named virtualenvwrapper
virtualenvwrapper.sh: There was a problem running the initialization hooks.

If Python could not import the module virtualenvwrapper.hook_loader,
check that virtualenvwrapper has been installed for
VIRTUALENVWRAPPER_PYTHON=/usr/bin/python and that PATH is
set properly.
`

Feature Request - crop image

@GuidoBartoli Apologies for the plethora of issues.....

A crop operation would be quite useful. This would involve drawing a box on the original image so that all subsequent operations operate on the masked region rather than the whole image. Would that be possible?

Features Roadmap

Is there a roadmap for the "italicized" features, or could there be one?

Thanks!

FAILED TO RUN THE PROGRAM

(sq) ┌──(kali㉿kali)-[~/Desktop/sherloq/gui]
└─$ python sherloq.py
Traceback (most recent call last):
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "/usr/lib/python3.9/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/usr/lib/python3.9/imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: /home/kali/.virtualenvs/sq/lib/python3.9/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: invalid ELF header

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/kali/Desktop/sherloq/gui/sherloq.py", line 34, in
from splicing import SplicingWidget
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/home/kali/Desktop/sherloq/gui/splicing.py", line 12, in
from noiseprint.noiseprint import genNoiseprint
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/home/kali/Desktop/sherloq/gui/noiseprint/noiseprint.py", line 16, in
import tensorflow.compat.v1 as tf
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/tensorflow/init.py", line 24, in
from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/tensorflow/python/init.py", line 49, in
from tensorflow.python import pywrap_tensorflow
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 142, in _import
return original_import(name, *args, **kwargs)
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "/home/kali/.virtualenvs/sq/lib/python3.9/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "/usr/lib/python3.9/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/usr/lib/python3.9/imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: /home/kali/.virtualenvs/sq/lib/python3.9/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: invalid ELF header

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.

Copy-Move Forgery

For a wider distribution compiled versions would be greatly appreciated.

Btw the MacOS link in the readme is outdated, I would link to https://dev.to/micuffaro/easy-workflow-for-switching-python-virtual-environments-with-zsh-19lc

There is one installation error

File "/Applications/sherloq/gui/digest.py", line 5, in <module> import magic
File "/Users/my/.virtualenvs/sherloq/lib/python3.7/site-packages/magic.py", line 201, in <module>
    raise ImportError('failed to find libmagic.

going back from 0.4.18 to 0.4.14 helped

(sherloq) (base) my@gui % pip uninstall python-magic-bin
(sherloq) (base) my@gui % pip install python-magic-bin==0.4.14

Great working, looking for the cloning tool

mark

radiusMatch

Would be interested to learn why cloning.py uses radiusMatch and not the more frequently seen knnMatch (with detection of inliners)? And what about block-based detection instead of keypoint-based? Even after careful adjustment of all parameter, I usually miss 50% of all CMDFs ...

Output formats?

Is there - or could there be - a way to output formats other than PNG for processed images, e.g. JPEG, PDF, .npy..? Thanks!

PRNU comparison doesn't work?

All the functions
in italics are not working?
Am I'm missing something?
Thanx

[Mac OS] Libmagic missing

On Mac OS Catalina, libmagic cannot be found when launching Sherloq

ImportError: failed to find libmagic.  Check your installation

Fixed by installing libmagic with brew

brew install libmagic

Segmentation fault while opening Hex Editor

Environment

Ubuntu 18.04.4 LTS
Python 3.7.6

uname output

uname -a
Linux aleryc-pc 5.3.0-62-generic #56~18.04.1-Ubuntu SMP Wed Jun 24 16:17:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Installed Packages (venv)

absl-py==0.9.0
astor==0.8.1
astunparse==1.6.3
cachetools==4.1.1
certifi==2020.6.20
chardet==3.0.4
concurrent-iterator==0.2.6
cycler==0.10.0
decorator==4.4.2
gast==0.3.3
google-auth==1.19.1
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.30.0
h5py==2.10.0
idna==2.10
imageio==2.9.0
importlib-metadata==1.7.0
joblib==0.16.0
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.2
kiwisolver==1.2.0
lxml==4.5.2
Markdown==3.2.2
matplotlib==3.2.2
networkx==2.4
numpy==1.19.0
oauthlib==3.1.0
opencv-contrib-python-headless==4.3.0.36
opt-einsum==3.2.1
pandas==1.0.5
Pillow==7.2.0
protobuf==3.12.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==2.4.7
PySide2==5.15.0
python-dateutil==2.8.1
python-magic==0.4.18
pytz==2020.1
PyWavelets==1.1.1
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.6
scikit-image==0.17.2
scikit-learn==0.23.1
scipy==1.5.1
sewar==0.4.3
shiboken2==5.15.0
six==1.15.0
tensorboard==2.2.2
tensorboard-plugin-wit==1.7.0
tensorflow==2.2.0
tensorflow-estimator==2.2.0
termcolor==1.1.0
threadpoolctl==2.1.0
tifffile==2020.7.4
urllib3==1.25.9
Werkzeug==1.0.1
wrapt==1.12.1
xgboost==1.1.1
zipp==3.1.0

Console Output

WARNING:tensorflow:From /home/aleryc/.virtualenvs/sq/lib/python3.7/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
[1]    25858 segmentation fault (core dumped)  python3 sherloq.py

Step to reproduce issue

1. Load Image (I only tried with PNG and JPEG).
2. Double click on Hex Editor in TOOLS tab

Thank you

Geolocation data tool

No matter what the image is, the geolocation data tool always points to the same place in Boston.

Python3.8 not supported

Just a heads up, there is a requirement issue with python3.8, as Tensorflow==1.15.3 is not supported. Well, Tensorflow < 2 isn't supported on python 3.8. May be worth a note in the README for others who run into the same issue. Downgrading to python 3.7 works fine.

Feature Request - support for multiple images open simultaneously

This would be super useful. Is it straightforward to implement or would it require major architectural changes?

Blink between Original and Processed images

Sherloq is amazing - thank you.

That aside - it would be very useful to have a 'blink' button that automatically toggles between overlaid Original and Processed images for a given operation at a specified or preset time interval. It would help enormously for comparing areas of interest to the original. Does that make sense?

This is borrowed from SAOIMAGE ds9 - https://sites.google.com/cfa.harvard.edu/saoimageds9

There's tonnes of other stuff that could be borrowed - including output of contours/regions of interest into ascii files that can be reloaded/overlaid etc., (x,y) coordinates for mouse, ...

Is there a rotate button anywhere?

Thanks again!

Containerization

Is there any merit - or is it even a good fit - to developing a Dockerfile for sherloq (or does one exist already?)?

See e.g https://stackoverflow.com/questions/60955960/how-to-write-a-dockerfile-to-run-python3-pyqt5

Bash problem

removed

Feature Request: Masking for Copy-Move Forgery Tool

Blatant feature request here. The ability to mask certain areas from being processed by the CMF tool would be wonderful. I am trying to use it for detecting image duplication in published articles (it's performing quite well!), and text in the figures result in a lot of noise. Being able to mask them within the program rather than editing beforehand would be very nice.

Bugs in Windows

Hi Guido, i installed on my windows laptop Sherloq. It was no easy because i crush into the "libmagic issue". At the end i solved installing matplotlib with pip. Now i got a serie of bugs when i click on some tools. First, when i start the software from the terminal i got the "error" that you can see in the image:
I'll give you the list of the tools that work and the tools that doesn't work;

1. File Digest - Doesn't work (in terminal appear an error "magic.MagicException: None") THIS ERROR DOES NOT CRASH THE SW
2. Hex Editor - Works well
3. Similiar Search - Works well (Could be nice have Yandex Reverse Image Search on the website list..)
4. Header Structure - Doesn't work (it's requested the "Enter" pressing inside the terminal, after you did it a windows in SW appear but with "impossible reach the website" info) THIS ERROR DOES NOT CRASH THE SW
5. EXIF Full Dump - Works well
6. Thumbnail Analysis - Works well (But it's requested the "Enter" pressing inside the terminal windows"
7. Geolocation Data - Not tested with geolocated picture, but seems to work
8. Enhancing Magnifier - Works well
9. Cannel Histogram - Works well
10. Global Adjustments - Works well
11. Reference Comparison - Works well
12. Luminance Gradient - Works well
13. Echo Edge Filter - Works well
14. Wavelet Treshold - Works well
15. Frequency Split - Works well
16. PCA Projection - Works well
17. Pixel Statitics - Works well
18. Noise Separation - Works well
19. MinMax Deviation - Works well
20. BitPlane Values - Works well
21. PRNU Identification - Doesnt Work (Nothing Happen when i press it) THIS ERROR DOES NOT CRASH THE SW
22. Quality Esimation - Doesnt Work THIS ERROR CRASH THE SW (No info appear in terminal windows)
23. Error Level Analysis - Works well
24. Multiple Compression - Doesnt Work (Nothing Happen when i press it) THIS ERROR DOES NOT CRASH THE SW
25. Jpeg Ghost Map - Doesnt Work (Nothing Happen when i press it) THIS ERROR DOES NOT CRASH THE SW
26. Contrast Enhancement - Works well
27. CopyMove Forgery - Works well
28. Composite Splicing - Works well
29. Image Resampling - Doesnt Work (Nothing Happen when i press it) THIS ERROR DOES NOT CRASH THE SW
30. Median Filtering - Works well
31. Illuminant Map - Doesnt Work (Nothing Happen when i press it) THIS ERROR DOES NOT CRASH THE SW
32. DeadHot Pixels - Doesnt Work (Nothing Happen when i press it) THIS ERROR DOES NOT CRASH THE SW
33. Stereogram Decoder - Works well

I'm really interested in the software, I have not particular computer skills, but if you need i could help you as a tester. Thanks for your work and trust me, is really interesting and if you improve it you could get big satisfaction from it

Frequent attribute errors in CloningWidget

Getting these every time the CMF tool is run:

Traceback (most recent call last):
  File "/mnt/c/Users/Jared/Documents/ImageDups/sherloq/gui/cloning.py", line 148, in cancel
    self.reset()
AttributeError: 'CloningWidget' object has no attribute 'reset'

Doesn't really appear to affect function, though I'm finding it pretty easy to freeze up the tool with larger/more complex images.

AttributeError: 'tuple' object has no attribute 'append'

When opening a JPEG file and selecting the channel histogram, the following error occurs;
Traceback (most recent call last): File "C:\Tools\Sherloq\sherloq\gui\sherloq.py", line 349, in open_tool tool_widget = HistWidget(self.image) File "C:\Tools\Sherloq\sherloq\gui\histogram.py", line 47, in __init__ channels.append(cv.cvtColor(image, cv.COLOR_BGR2GRAY)) AttributeError: 'tuple' object has no attribute 'append'

I am on Windows 10 Professional, Python 3.9.6, and Sherloq 0.85a

Quality estimation

Why i can't use quality estimation? It always comes to error (not responding) and then freeze orang force closing. Thank you

Increase the usage of augmented assignment statements

👀 Some source code analysis tools can help to find opportunities for improving software components.
💭 I propose to increase the usage of augmented assignment statements accordingly.

Would you like to integrate anything from a transformation result which can be generated by a command like the following?
(:point_right: Please check also for questionable change suggestions because of an evolving search pattern.)

lokal$ perl -p -i.orig -0777 -e 's/^(?<indentation>\s+)(?<target>\S+)\s*=\s*\k<target>[ \t]*(?<operator>[+%^@]|-(?!>)|&(?!&)|\|(?!\|)|\*\*?|\/\/?|<<|>>)/$+{indentation}$+{target} $+{operator}=/gm' $(find ~/Projekte/Sherloq/lokal -name '*.py')