joshhighet / transl8

a common schema for internet-connected service discovery 🔎 📚

transl8

transl8 is a syntax translator and query generator for network search engines. it can help compile cross-platform queries for various global network scan data providers.

the web service is available at transl8.watchdawg.io

there are two CSV's which define the data presented on the above site;

• queries.csv - the query lookup table
• providers.csv - the search provider configuration table

whern proposing query changes, PR one of the above - a GitHub action will automatically update the lookups leveraged by the web service.

Platform	Login Location	Query Documentation
ZoomEye	zoomeye.org/login	zoomeye.org/doc
Censys	censys.io/login	search.censys.io/search/definitions
FOFA Search	en.fofa.info/f_login	en.fofa.info/api
Quake 360	quake.360.net/quake/login#/	quake.360.net/quake/#/help
Netlas	app.netlas.io/login/	app.netlas.io/responses/
Onyphe	onyphe.io/signin	onyphe.io/docs
Criminal IP	criminalip.io/en/login	criminalip.io/en/developer
Hunter	hunter.how/profile	hunter.how/guide
Shodan	account.shodan.io/login	beta.shodan.io/search/filters
BinaryEdge	app.binaryedge.io/login	docs.binaryedge.io/search
urlscan.io	urlscan.io/user/login/	urlscan.io/docs/search/
FullHunt	fullhunt.io/login	fullhunt.io/docs/search/filters

transl8's Issues

docs link not clickable if query’s unavailable

There is no AND syntax in ZoomEye

There is no syntax such as AND OR in ZoomEye. AND conditions are represented by + and OR conditions are represented by spaces.
for example:
ip:127.0.0.1 + port:53
ip:127.0.0.1 ip:127.0.0.2

Looks Funny 😜

Consider reading CSS For Dummies

add new zoomeye domain (.hk)

zoom eye: multiple query requires +

I.e iconhash:"$fieldstr" AND +country:"US"

netlas: multiple queries requires ()

i.e (http.title:PaperCut) AND geo.country:("NZ")

when selecting country a number of search providers disappear

they should be greyed out like other field types, I.e

instead they disappear. likely because country field input has a extension function to correlate country code to country name for some providers that require the full string name over ISO code

add fullhunt

fullhunt.io/docs/search/filters/

link into urlgen project from homepage

https://transl8.watchdawg.io/urlgen/

add asn search prefix context

some platforms expect asn numerical searches to be prefixes with AS whilst others don't

dual input fields on first load

build constructed query into URI query params

allow for a query to be shared by link

add: `last modified` http header query

binaryedge
web.headers.last_modified:"Sun, 27 Aug 2023 16:25:16 GMT"

hunter.how
header="Sun, 27 Aug 2023 16:25:16 GMT"

netlas
http.headers.last_modified:"Sun, 27 Aug 2023 16:25:16 GMT"

censys
services.http.response.headers.Last_Modified: "Sun, 27 Aug 2023 16:25:16 GMT"

if query value 'freeform' - do not append `freeform:`

the CSV value set as "freeform" means the value is able to be searched but not against an operator

add greynoise

https://viz.greynoise.io/cheat-sheet

fofa: queries with more than one keyword require encoding

see the base64 param

add urlscan

https://urlscan.io/search/# && spec ref https://urlscan.io/docs/search/

use providers record

remove js consts and allow multi-query operators

[
  {
    "name": "shodan",
    "prefix": "https://www.shodan.io/search?query=",
    "docs": "https://beta.shodan.io/search/filters",
    "kv_separator": ":",
    "query_separator": null,
    "operators/and": "and",
    "operators/or": "or",
    "operators/not": "not",
    "as_includeprefix": "TRUE",
    "png_uri": "https://pbs.twimg.com/profile_images/1105606704090267648/oyZUgnFr_400x400.png"
  },
  {
    "name": "censys",
    "prefix": "https://search.censys.io/search?resource=hosts&sort=RELEVANCE&q=",
    "docs": "https://search.censys.io/search/definitions?resource=hosts",
    "kv_separator": ":",
    "query_separator": " and ",
    "operators/and": "and",
    "operators/or": "or",
    "operators/not": "not",
    "as_includeprefix": "FALSE",
    "png_uri": "https://seeklogo.com/images/C/censys-logo-F1BB7A3D1D-seeklogo.com.png"
  }
]

add hunter.how

https://hunter.how/guide

enforce input constraints/tips

where a rex exists, give some kind of tip on inputs failing a match if constraint has been populated

[
  {
    "keyword": "country",
    "shodan": "country",
    "shodan-images": "country",
    "censys": "location.country_code",
    "binaryedge": "country",
    "zoomeye": "country",
    "fofa": "country",
    "quake360": "country",
    "netlas": "geo.country",
    "onyphe": "country",
    "crimninalip-images": null,
    "criminalip": "country",
    "description": "Country Code (ISO)",
    "constraint": "^[A-Za-z]{2}$",
    "example": "NZ"
  }
]

criminalip: use quotes

a query of ip:"1.1.1.1" returns nothing while a query for ip:1.1.1.1 will return data
however a search for port:53 and port:"53" will return the same results
the current no-quote ip address query means you will receive search errors
https://www.criminalip.io/en/mypage/support/647

open all links button

censys - and seperator present regardless of keyword count

adhere to separators

relies on #3

not all platforms use the same key/value separator or the same search operators and default spacing terminology

leverage what is available at a per-provider level to construct queries properly

"kv_separator": ":",
"query_separator": null,
"operators/and": "and",
"operators/or": "or",
"operators/not": "not",
"as_includeprefix": "TRUE",

quake360: country needs to be full name not ISO

a ISO to country lookup will need to be created - ISO codes can not be used

i.e title: "Login" AND country: "United States"