When you need to create complex Workflows and need to communicate different tools working together, maybe you need MIST
.
MIST
is a high level programming language for defining executions workflows easily.
Ab execution is a command line tool you can invoke from MIST
. They will connect the tools and manage executions and synchronization fo you.
> pip install mist-lang
Explanation
In this scenario we'll do:
CLI Input
- Read a domain as a parameter from CLI.Search Domains
- Use MIST function for search related domains / sub-domains from a start domain.Fin OpenPorts
- Search open port for each new domain / sub-domain found.Screen (Pring)
- Displays the results into the screen (by using MIST 'print' function).
Use case diagram
MIST code (examples/demo/scenario-01.mist)
include "searchDomains" include "findOpenPorts"
searchDomains(%domain) => findOpenPorts("80,443") => print()
Execute
> mist examples/demo/scenario-01.mist domain=example.com
Explanation
In this scenario we'll do:
CLI Input
- Read a domain as a parameter from CLI.Search Domains
- Use MIST function for search related domains / sub-domains from a start domain.FindOpenPorts
- Search open port for each new domain / sub-domain found.Kafka output
- Send results to a Kafka topic.
Use case diagram
MIST code (examples/demo/scenario-02.mist)
include "searchDomains" "findOpenPorts" "kafkaProducer"
searchDomains(%domain) => findOpenPorts("80,443") =>
kafkaProducer($KAFKA_SERVER, "domainsTopic")
Execute
> mist examples/demo/scenario-02.mist domain=example.com
Explanation
In this scenario we'll do:
CLI Input
- Read a domain as a parameter from CLI.- Search domains:
Search Domains
- Use MIST function for search related domains / sub-domains from a start domain.Festin
- Use MIST integration for Festin for search related domains / sub-domains from a start domain.
Filter Repeated
- Use MIST function to detect and remove repeated found domains.Fin OpenPorts
- Search open port for each new domain / sub-domain get fromFitler Repeated
.Kafka output
- Send results to a Kafka topic.
Use case diagram
MIST code (examples/demo/scenario-03.mist)
include "searchDomains" "festin" "findOpenPorts" "filterRepeated" "kafkaProducer"
searchDomains(%domain) => foundDomains
festin(%domain, $DNS_SERVER, True) => foundDomains
foundDomains => filterRepeated(False) =>
findOpenPorts("80,443") => kafkaProducer($KAFKA_SERVER, "domainsTopic")
Execute
> mist examples/demo/scenario-03.mist domain=example.com
Explanation
In this scenario we'll do:
CLI Input
- Read a domain as a parameter from CLI.- Search domains:
Search Domains
- Use MIST function for search related domains / sub-domains from a start domain.Festin
- Use MIST integration for Festin for search related domains / sub-domains from a start domain.
Filter Repeated
- Use MIST function to detect and remove repeated found domains.Find OpenPorts
- Search open port for each new domain / sub-domain get fromFitler Repeated
.Dispatcher (80 / 443)
- Split results and send each port to a different queue.- Send results:
Kafka output
- Send found 80 ports to a Kafka topic.S3 output
- Send found 443 ports to a AWS S3 bucket.
Use case diagram
MIST code (examples/demo/scenario-04.mist)
include "searchDomains" "festin" "findOpenPorts" "filterRepeated" "kafkaProducer" "S3Store"
function dispatcher(p) => kafka, S3 {
if (isEqual(p.port, "80")) {
p => kafka
} else {
p => S3
}
}
searchDomains(%domain) => foundDomains
festin(%domain, $DNS_SERVER, True) => foundDomains
foundDomains => filterRepeated(False) =>
findOpenPorts("80,443") => dispatcher() => kafkaOutput, S3Output
kafkaOutput => kafkaProducer($KAFKA_SERVER, "domainsTopic")
S3Output => S3Store($BUCKET_URI)
Execute
> mist examples/demo/scenario-04.mist domain=example.com
Explanation
In this scenario we'll do:
1 Input from multiple sources:
File Input
- Read domains from an external file.Kafka Input
- Read domains from Kafka topics.CLI Input
- Read domains from CLI.- Search domains:
Search Domains
- Use MIST function for search related domains / sub-domains from a start domain.Festin
- Use MIST integration for Festin for search related domains / sub-domains from a start domain.
Filter Repeated
- Use MIST function to detect and remove repeated found domains.Find OpenPorts
- Search open port for each new domain / sub-domain get fromFitler Repeated
.Dispatcher (80 / 443)
- Split results and send each port to a different queue.- Send results:
Kafka output
- Send found 80 ports to a Kafka topic.S3 output
- Send found 443 ports to a AWS S3 bucket.
Use case diagram
MIST code (examples/demo/scenario-05.mist)
include "searchDomains" "festin" "findOpenPorts" "filterRepeated" "kafkaProducer" "S3Store" "kafkaConsumer" "tail"
function dispatcher(p) => kafka, S3 {
if (isEqual(p.port, "80")) {
p => kafka
} else {
p => S3
}
}
kafkaConsumer($KAFKA_SERVER, "inputTopic", "*END*", False) => inputDomains
tail("domains.txt", "*END*") => inputDomains
%domain => inputDomains
inputDomains => searchDomains() => foundDomains
inputDomains => festin($DNS_SERVER, True) => foundDomains
foundDomains => filterRepeated(False) => findOpenPorts("80,443") =>
dispatcher() => kafkaOutput, S3Output
kafkaOutput => kafkaProducer($KAFKA_SERVER, "domainsTopic")
S3Output => S3Store($BUCKET_URI)
Execute
> mist examples/demo/scenario-05.mist domain=example.com
MIST is being developed by BBVA-Labs Security team members.
Contributions are of course welcome. See CONTRIBUTING or skim existing tickets to see where you could help out.
MIST is Open Source Software and available under the Apache 2 license