cybercentrecanada / assemblyline-service-badlist

Assemblyline 4 service for badlisting network and file features

Badlisting service

This service allow you to mark bad a set of files and tags so they always score as malicious in Assemblyline. The content of the badlist is driven by badlist sources or user marked files or IOCs strait from Assemblyline's UI.

Sources

When adding sources to the service, there are two types of expected data formats

• csv
• json

There are also multiple types of sources for this service:

• blocklist
• malware_family_list
• attribution_list

Blocklist Data Formats

In order for the service to pull the right IOCs and categorize them per source, you'll have to instruct it on how to using the config.updater.<source> key.

Within each source map, you'll specify the type of source this is (blocklist) as well as set the format (json | csv).

You'll also have to specify the different IOC types (domain, ip, uri, md5, sha1, sha256, ssdeep, tlsh) you expect to find in the data and where.

For example if dealing with a CSV file and you expect to find uris in the 3rd column per row:

ie. "<date>,<name>,https://google.com,..."

Then your source configuration will look like:

config:
  updater:
    my_source:
      type: blocklist
      format: csv
      uri: 2

Similarly, if you're dealing with a JSON list ([{}, {}, ...]) and you know to find uris under the key bad_uri in each record:

ie. {"bad_uri": "https://google.com", "family": "bad_stuff", ...}

config:
  updater:
    my_source:
      type: blocklist
      format: json
      uri: "bad_uri"

You can also override Assemblyline's default scoring of badlist matches (1000 points) by providing a score per source.

Automated Expiration

By default, we assume that all the items added to the Badlist will be valid forever but that's not always the cases. You will also be able to set a DTL (Days to Live) period for items that belong to a source using dtl.

If there are multiple sources with DTLs configured that raise an item, then the expiry date will be extended by the sum of the DTL values at the time of importing.

assemblyline-service-badlist's Issues

IOC and IOC Type are switched around.

assemblyline-service-badlist/badlist/badlist.py

Line 122 in 0ff26ee

"IOC": badlisted["tag"]["type"],

Is currently;

                    body={
                        "IOC": badlisted["tag"]["type"],
                        "IOC Type": badlisted["tag"]["value"],
                        "First added": badlisted["added"],
                        "Last updated": badlisted["updated"],
                    },

Should be;

                    body={
                        "IOC": badlisted["tag"]["value"],
                        "IOC Type": badlisted["tag"]["type"],
                        "First added": badlisted["added"],
                        "Last updated": badlisted["updated"],
                    },

Updater Service performance

Description

Badlist updater service seems quite slow, for example the default provided Threatfox updater takes ~30 minutes to complete.
I'm assuming here that it continues to the next update source immediately after finishing Threatfox.

{"@timestamp": "2024-01-29 04:23:18,285", "event": { "module": "assemblyline", "dataset": "assemblyline.updater.badlist" }, "host": { "ip": "10.233.89.56", "hostname": "alsvc-badlist-updates-6f7c5c979-fq2p8" }, "log": { "level": "INFO", "logger": "assemblyline.updater.badlist" }, "process": { "pid": "1" }, "message": "threatfox source is configured to verify SSL."}
{"@timestamp": "2024-01-29 04:23:22,938", "event": { "module": "assemblyline", "dataset": "assemblyline.updater.badlist" }, "host": { "ip": "10.233.89.56", "hostname": "alsvc-badlist-updates-6f7c5c979-fq2p8" }, "log": { "level": "INFO", "logger": "assemblyline.updater.badlist" }, "process": { "pid": "1" }, "message": "Found new badlist rule files to process for threatfox!"}
{"@timestamp": "2024-01-29 04:56:02,625", "event": { "module": "assemblyline", "dataset": "assemblyline.updater.badlist" }, "host": { "ip": "10.233.89.56", "hostname": "alsvc-badlist-updates-6f7c5c979-fq2p8" }, "log": { "level": "INFO", "logger": "assemblyline.updater.badlist" }, "process": { "pid": "1" }, "message": "threatview.io_C2_Hunt_Feed source is configured to verify SSL."}

Also have a CSV of 70k IPs that takes 6-7 minutes each time to import.

{"@timestamp": "2024-01-29 05:15:10,742", "event": { "module": "assemblyline", "dataset": "assemblyline.updater.badlist" }, "host": { "ip": "10.233.89.56", "hostname": "alsvc-badlist-updates-6f7c5c979-fq2p8" }, "log": { "level": "INFO", "logger": "assemblyline.updater.badlist" }, "process": { "pid": "1" }, "message": "threatdata_ipv4 source is configured to verify SSL."}
{"@timestamp": "2024-01-29 05:15:10,946", "event": { "module": "assemblyline", "dataset": "assemblyline.updater.badlist" }, "host": { "ip": "10.233.89.56", "hostname": "alsvc-badlist-updates-6f7c5c979-fq2p8" }, "log": { "level": "INFO", "logger": "assemblyline.updater.badlist" }, "process": { "pid": "1" }, "message": "Found new badlist rule files to process for threatdata_ipv4!"}
{"@timestamp": "2024-01-29 05:22:12,637", "event": { "module": "assemblyline", "dataset": "assemblyline.updater.badlist" }, "host": { "ip": "10.233.89.56", "hostname": "alsvc-badlist-updates-6f7c5c979-fq2p8" }, "log": { "level": "INFO", "logger": "assemblyline.updater.badlist" }, "process": { "pid": "1" }, "message": "threatdata_domains source is configured to verify SSL."}

Probably not the biggest issue, I only really noticed it while debugging an import source as I've got to wait for it to iterate over each of update sources before it'll have another go at the one I'm making changes to.
Didn't really notice this when doing Netrep updates.

Steps to Reproduce

Run the updater with defaults and see how long it takes.

Expected behavior:
Runs significantly faster

Actual behavior:
Takes significant time to run, also appears to take the same amount of time on subsequent imports.

Versions

AssemblyLine - Stable90
assemblyline-service-badlist - 4.4.stable4