soxoj / marple

📖 Collect links to profiles by username through search engines and analyze with various plugins

Marple

Summary

Collect links to profiles by username through 10+ search engines (see the full list below).

Features:

• multiple engines
• proxy support
• CSV file export
• plugins
  • pdf metadata extraction
  • social media info extraction

Quick Start

./marple.py soxoj

Results:

https://t.me/soxoj
Contact @soxoj - Telegram

https://github.com/soxoj
soxoj - GitHub

https://coder.social/soxoj
soxoj - Coder Social

https://gitmemory.com/soxoj
soxoj

...

PDF files
https://codeby.net/attachments/v-0-0-1-social-osint-fundamentals-pdf.45770
Social OSINT fundamentals - Codeby.net
/Creator: Google

...

Links: total collected 111 / unique with username in URL 97 / reliable 38 / documents 3

Advanced usage:

./marple.py soxoj --plugins metadata

./marple.py smirnov --engines google baidu -v

Installation

All you need is Python3. And pip. And requirements, of course.

pip3 install -r requirements.txt

You need API keys for some search engines (see requirements in Supported sources). Keys should be exported to env in this way:

export YANDEX_KEY=key

Options

You can specify 'junk threshold' with option -t or --threshold (default 300) to get more or less reliable results.

Junk score is summing up from length of link URL and symbols next to username as a part of URL.

Also you can increase count of results from search engines with option --results-count (default 1000). Currently limit is only applicable for Google.

Other options:

  -h, --help            show this help message and exit
  -t THRESHOLD, --threshold THRESHOLD
                        Threshold to discard junk search results
  --results-count RESULTS_COUNT
                        Count of results parsed from each search engine
  --no-url-filter       Disable filtering results by usernames in URLs

  --engines {baidu,dogpile,google,bing,ask,aol,torch,yandex,naver,paginated,yahoo,startpage,duckduckgo,qwant}
                        Engines to run (you can choose more than one)

  --plugins {socid_extractor,metadata,maigret} [{socid_extractor,metadata,maigret} ...]
                        Additional plugins to analyze links

  -v, --verbose         Display junk score for each result
  -d, --debug           Display all the results from sources and debug messages
  -l, --list            Display only list of all the URLs
  --proxy PROXY         Proxy string (e.g. https://user:[email protected]:8080)
  --csv CSV             Save results to the CSV file

Supported sources

Name	Method	Requirements
Google	scraping	None, works out of the box; frequent captcha
DuckDuckGo	scraping	None, works out of the box
Yandex	XML API	Register and get YANDEX_USER/YANDEX_KEY tokens
Naver	SerpApi	Register and get SERPAPI_KEY token
Baidu	SerpApi	Register and get SERPAPI_KEY token
Aol	scraping	None, scrapes with pagination
Ask	scraping	None, scrapes with pagination
Bing	scraping	None, scrapes with pagination
Startpage	scraping	None, scrapes with pagination
Yahoo	scraping	None, scrapes with pagination
Mojeek	scraping	None, scrapes with pagination
Dogpile	scraping	None, scrapes with pagination
Torch	scraping	Tor proxies (socks5://localhost:9050 by default), scrapes with pagination
Qwant	Qwant API	Check if search available in your exit IP country, scrapes with pagination

Development & testing

$ python3 -m pytest tests

TODO

• Proxy support
• Engines choose through arguments
• Exact search filter
• Engine-specific filters
• 'Username in title' check

Mentions and articles

Sector035 - Week in OSINT #2021-50

OS2INT - MARPLE: IDENTIFYING AND EXTRACTING SOCIAL MEDIA USER LINKS

marple's Issues

Add seznam.cz

https://search.seznam.cz/

Add checks for PDF mime type while downloading

Add Ecosia search engine

Add Mock to requirements.txt

index.html

Add Petal

https://petalsearch.com/

Ошибка No module named 'yandex_search'

Вот такую ошибку выдает:

Traceback (most recent call last):
File "/home/panther/marple/marple.py", line 13, in
import yandex_search
ModuleNotFoundError: No module named 'yandex_search'

add termcolor to requirements

Traceback (most recent call last):
File "marple.py", line 4, in
from termcolor import colored
ModuleNotFoundError: No module named 'termcolor'

Remove links to image search

[117] [Google] /search?q=...
Images

Add discrete search

https://www.discretesearch.com/

ImportError on first use

git clone https://github.com/soxoj/marple.git
cd marple
pip3 install -r requirements.txt

$ ./marple.py soxoj
Traceback (most recent call last):
File "./marple.py", line 15, in
from aiohttp_socks import ProxyConnector
File "/home/[]/.local/lib/python3.8/site-packages/aiohttp_socks/init.py", line 5, in
from .connector import (
File "/home/[]/.local/lib/python3.8/site-packages/aiohttp_socks/connector.py", line 8, in
from aiohttp.helpers import CeilTimeout # noqa
ImportError: cannot import name 'CeilTimeout' from 'aiohttp.helpers' (/home/[*******]/.local/lib/python3.8/site-packages/aiohttp/helpers.py)

System: Linux Mint Xfce on Proxmox

Add Gibiru search engine

https://gibiru.com/

Add searx

https://github.com/searx/searx-instances/blob/master/searxinstances/instances.yml

add aiohttp to requirements

Traceback (most recent call last):
File "marple.py", line 9, in
import aiohttp
ModuleNotFoundError: No module named 'aiohttp'

Maybe add first/last name patterns generator like NAMINT?

https://seintpl.github.io/NAMINT/

Ability to do additional searches with first/last name

Add notEvil

Add support of nicknames with special symbols

https://discord.com/channels/519965153559576590/519993240598872064/911538298206445609

-_'".

Create a directory with username for PDF files

ImportError: cannot import name 'Aol' from 'search_engines'

When running:

./marple.py [my_target_username]

I get:

Traceback (most recent call last):

File "/home/osint/Programas/marple/./marple.py", line 21, in

from search_engines import Aol, Ask, Qwant, Bing, Yahoo, Startpage, Dogpile, Mojeek, Torch, Duckduckgo

ImportError: cannot import name 'Aol' from 'search_engines' (/usr/local/lib/python3.10/dist-packages/search_engines/init.py)