hartl3y94 / analyzer

Analyze, extract and visualize features, artifacts and IoCs of files and memory dumps (Windows, Linux, Android, iPhone, Blackberry, macOS binaries, emails and more)

This project automates the daily tasks of Threat Intelligence Analyzer role internally without external resources' interaction. It analyzes, visualizes and structures sensitive files (Malware) or data by extracting features, artifacts and IoC using different modules. The output of those modules can be easily integrated in your research or SOC platforms.

Install

git clone https://github.com/qeeqbox/analyzer.git && cd analyzer && chmod +x run.sh && ./run.sh auto_configure

Interface

Output

• APT-Malware JSON\HTML reports (+190 sample)

Features

• Runs locally (Offline)
• Analyze buffer, file or full folder
• Intime analysis (Session is saved)
• 2 modes (Interactive and silent)
• Generates HTML or JSON as output
• Dump output file with details to mongodb
• Save raw json result to mongodb
• Basic file information MD5, charset, mime, ssdeep
• Different string/patterns analysis methods
• Web service and API
• Ports, IPS hints and countries description
• World IPS world image and flags
• DNS servers description (Top servers)
• Artifacts force directed image
• Cross references force directed image and table
• Similarity image divided to classes
• YARA module, and YARA rules from yara-rules-github)
• YARA module includes conditions & tags by index
• Whitelist implemented (Windows7, 8 and 10 files)
• Check WAF and bypass proxy
• Spelling and punctuation check
• Top phishing words included
• Snort support
• PDF, RTF, Phishing, MS office and HTML modules
• NL English words detection
• OCR words detection
• Websites similarity detection (Top 10000)
• BOM (Byte Order Mark) detection
• MITRE att&ck tools and patterns detection (could be FP)
• URL shorteners extraction
• ASCII from UNICODE extraction
• Free\Fake email extraction
• URL, EMAIL, and TEL Tags patterns extraction
• Credit Cards, Credential, and Secrets patterns patterns extraction
• Encryption patterns (base64, md5, sha1..) extraction
• DGA (Domain Generation Algorithm) patterns extraction

Modules

• Linux wrapper - ELF information, API functions descriptions, System commands descriptions, Sections descriptions, Lib descriptions, Encrypted section detection, Symbols extraction, MITRE artifacts mapped to detection, Cross references detection, Behavior detection
• Windows wrapper - PE information, Encrypted section detection, Sections descriptions, DLL descriptions, Symbols extraction, Signature extraction and validation, API descriptions, PE ASLR, DEP, SEH and CFG detection, MITRE artifacts mapped to detection, API Behavior detection, DLL injection, Process Hollowing, Process Doppelganging etc.., Cross references detection, Icon extraction, Extract String file info, FileDescription, FileDescription etc..
• Android wrapper - APK information, DEX information, Manifest descriptions, Intent descriptions, Resources extraction, Symbols extraction, Classes extraction, Big functions identification, Cross references detection, API Behavior detection
• IPhone built-in - IPA information
• BlackBerry COD built-in - COD information, Functions extraction, Strings extraction
• PCAP wrapper - Frame filter, HTTP filter, DNS filter, ARP filter, WAF detection, DGA detection, Snort parsing
• PDF built-in - Objects enumeration, Keys, javascript, js, OpenAction, extraction, Streams parsing, String analysis
• Office built-in and wrapper - Meta info extraction, Hyper and target links extraction, Bin printable parser, Extract Text, Extract DDE, Macros extraction
• OLE wrapper - Number of objects, Object extraction, Macros extraction
• EMAIL built-in and wrapper - Header information, Attachment extraction and parsing, Extract body, Phishing patterns check
• Archives wrapper - Extract mimes and guess by extensions, Finding patterns in all unpacked files, Encrypted archives detection
• HTML wrapper - Extract scripts, iframes, links and forms, Decode/analyze links, Script entropy
• Some patterns - AWS Clint ID, Amazon MWS Auth Token, Amazon S3, ALIYUN OSS, AZURE Storage, Facebook Access Token, Github Token, Goole API Key, Google CAPTCHA, Google OAuth, Google Secret, Google OAuth Access Token, Mailgun API Key, MailChimp API, Picatic API, Slack Token, Square Access Token, Square OAuth Secret, Stripe API, Twilio API, Twilio SID

One click auto-configure

git clone https://github.com/qeeqbox/analyzer.git
cd analyzer
chmod +x run.sh
./run.sh auto_configure
The project interface http://127.0.0.1:8000/login/ will open automatically after finishing the initialization process

Or, if you already have docker-compose

docker-compose -f docker-compose-dev.yml up --build
Then open http://127.0.0.1:8000/login/

Prerequisites

apt-get install -y python3 python3-pip curl libfuzzy-dev yara libmagic-dev libjansson-dev libssl-dev libffi-dev tesseract-ocr libtesseract-dev libssl-dev swig p7zip-full radare2 dmg2img mongodb redis

pip3 install pyelftools macholib python-magic nltk Pillow jinja2 ssdeep pefile scapy r2pipe pytesseract M2Crypto requests tld tldextract bs4 psutil pymongo flask pyOpenSSL oletools extract_msg

Prerequisites packages are required for some modules (If you are having issues using those packages, I might be able to share with you my own alternatives that I developed in the past in C#\C)

Roadmap

• Java analysis (Requested by users)
• Web detection
• Adding username and password wrappers to databases
• CSS clean up

Resources

Linux documentation, MacOS documentation, Windows documentation, Android documentation, software77, MITRE ATT&CK™, sc0ty, hexacorn, PEID, steren, bacde, cisco umbrella , yara rules community , TONS OF RESEARCHES

Other Licenses

By using this framework, you are accepting the license terms of all these packages: yara, Yara-Rules, tesseract-ocr, swig, radare, vu1tur, oletools, mongodb, supervisor, msg-extractor, snort, pyelftools, macholib, pefile, scapy, python-magic, flask, werkzeug, gunicorn, flask-mongoengine, flask-admin, flask-login, flask-bcrypt, pyopenssl, flask-markdown, tld, psutil, gevent, dateutil, requests, pymongo, BeautifulSoup, tldextract, m2crypto, radare2, ssdeep, jinja2, Pillow, nltk, p7zip, redislabs, redis-py

Disclaimer\Notes

• Do not deploy without proper configuration
• Setup some security group rules and remove default credentials
• This project is NOT an anti malware project and does not quarantine or delete malicious files
• This project was developed for analyzing classified data and training some AI locally without internet/external interaction
• Please let me know if i missed a resource or dependency

Other Projects