GithubHelp home page GithubHelp logo

eliasgranderubio / dagda Goto Github PK

View Code? Open in Web Editor NEW
1.1K 40.0 157.0 1.19 MB

a tool to perform static analysis of known vulnerabilities, trojans, viruses, malware & other malicious threats in docker images/containers and to monitor the docker daemon and running docker containers for detecting anomalous activities

License: Apache License 2.0

Python 99.46% Makefile 0.02% Shell 0.47% Dockerfile 0.06%
docker security static-analysis vulnerabilities detecting-anomalous-activities malware-detection

dagda's Introduction

Dagda

Python Docker Pulls License

Dagda is a tool to perform static analysis of known vulnerabilities, trojans, viruses, malware & other malicious threats in docker images/containers and to monitor the docker daemon and running docker containers for detecting anomalous activities.

In order to fulfill its mission, first the known vulnerabilities as CVEs (Common Vulnerabilities and Exposures), BIDs (Bugtraq IDs), RHSAs (Red Hat Security Advisories) and RHBAs (Red Hat Bug Advisories), and the known exploits from Offensive Security database are imported into a MongoDB to facilitate the search of these vulnerabilities and exploits when your analysis are in progress.

Then, when you run a static analysis of known vulnerabilities, Dagda retrieves information about the software installed into your docker image, such as the OS packages and the dependencies of the programming languages, and verifies for each product and its version if it is free of vulnerabilities against the previously stored information into the MongoDB. Also, Dagda uses ClamAV as antivirus engine for detecting trojans, viruses, malware & other malicious threats included within the docker images/containers.

Dagda supports multiple Linux base images:

  • Red Hat/CentOS/Fedora
  • Debian/Ubuntu
  • OpenSUSE
  • Alpine

Dagda rests on OWASP dependency check + Retire.js for analyzing multiple dependencies from:

  • java
  • python
  • nodejs
  • js
  • ruby
  • php

On the other hand, Dagda is integrated with Falco for monitoring running docker containers to detect anomalous activities. Also, Dagda includes the gathering of real time events from docker daemon.

Finally, each analysis report of a docker image/container, included all static analysis and all runtime monitoring, is stored into the same MongoDB for having available the history of each docker image/container when it is needed.

Requirements

Before Dagda usage, you must have installed the next requirements:

  • Python 3.8.X or later
  • MongoDB 3.6 or later
  • Docker
  • Pip3
    • PyMongo
    • Requests
    • Python-dateutil
    • Joblib
    • Docker
    • Flask
    • Flask-cors
    • PyYAML
    • Defusedxml
    • Waitress

The requirements can be installed with pip:

    sudo pip3 install -r requirements.txt

Installation of Docker

You must have installed Docker for using Dagda. If you need instructions for Docker installation, see the How-to install Docker page.

In order to avoid having to use sudo when you use the docker command, create a Unix group called docker and add users to it. When the docker daemon starts, it makes the ownership of the Unix socket read/writable by the docker group.

Installation of MongoDB

You must have installed MongoDB 3.6 or later for using Dagda because in MongoDB are stored both the vulnerabilities/exploits and the analysis results.

If you need instructions for MongoDB installation, see the How-to install MongoDB Community Edition page.

You can also run MongoDB using docker:

    docker pull mongo
    docker run -d -p 27017:27017 mongo

Installation of kernel headers in the host OS

You must have installed the kernel headers in the host OS because Dagda is integrated with Falco for monitoring running docker containers to detect anomalous activities.

This can usually be done on Debian-like distributions with: apt-get -y install linux-headers-$(uname -r)

Or, on RHEL-like distributions: yum -y install kernel-devel-$(uname -r)

After that, run the command /usr/lib/dkms/dkms_autoinstaller start is recommended for avoiding the next Sysdig Falco error trace:

rmmod: ERROR: Module sysdig_probe is not currently loaded

Important to note: In some distributions it has been detected that Sysdig installation is required, so if you need instructions for Sysdig installation, see the How-to install Sysdig for Linux page.

Usage

You must run python3 dagda.py start for starting the Dagda server. See the start sub-command in the wiki page for details.

After the Dagda server started and before the Dagda CLI usage, you must set the next environment variables as you need:

    export DAGDA_HOST='127.0.0.1'
    export DAGDA_PORT=5000

Although in this usage documentation only the CLI usage is shown, Dagda has a REST API for using it. See REST API documentation page for details.

Populating the database

For the initial run, you need to populate the vulnerabilities and the exploits in the database by running:

    python3 dagda.py vuln --init

The previous command can take several minutes for finishing so be patient.

If you need repopulating your database for updating with the new vulnerabilities and exploits, you only need rerun the previous command.

Also, you can run queries on your personal database with dagda.py vuln. A usage example would be the next one:

    python3 dagda.py vuln --product openldap --product_version 2.2.20

The expected output for the previous query is shown below:

    [
        {
            "CVE-2005-4442": {
                "cveid": "CVE-2005-4442",
                "cvss_access_complexity": "Low",
                "cvss_access_vector": "Local access",
                "cvss_authentication": "None required",
                "cvss_availability_impact": "Complete",
                "cvss_base": 7.2,
                "cvss_confidentiality_impact": "Complete",
                "cvss_exploit": 3.9,
                "cvss_impact": 10.0,
                "cvss_integrity_impact": "Complete",
                "cvss_vector": [
                    "AV:L",
                    "AC:L",
                    "Au:N",
                    "C:C",
                    "I:C",
                    "A:C"
                ],
                "cweid": "CWE-0",
                "mod_date": "05-09-2008",
                "pub_date": "20-12-2005",
                "summary": "Untrusted search path vulnerability in OpenLDAP before 2.2.28-r3 on Gentoo Linux allows local users in the portage group to gain privileges via a malicious shared object in the Portage temporary build directory, which is part of the RUNPATH."
            }
        },
        {
            "CVE-2006-2754": {
                "cveid": "CVE-2006-2754",
                "cvss_access_complexity": "Low",
                "cvss_access_vector": "Network",
                "cvss_authentication": "None required",
                "cvss_availability_impact": "None",
                "cvss_base": 5.0,
                "cvss_confidentiality_impact": "None",
                "cvss_exploit": 10.0,
                "cvss_impact": 2.9,
                "cvss_integrity_impact": "Partial",
                "cvss_vector": [
                    "AV:N",
                    "AC:L",
                    "Au:N",
                    "C:N",
                    "I:P",
                    "A:N"
                ],
                "cweid": "CWE-0",
                "mod_date": "07-03-2011",
                "pub_date": "01-06-2006",
                "summary": "Stack-based buffer overflow in st.c in slurpd for OpenLDAP before 2.3.22 might allow attackers to execute arbitrary code via a long hostname."
            }
        },
        {
            "CVE-2006-5779": {
                "cveid": "CVE-2006-5779",
                "cvss_access_complexity": "Low",
                "cvss_access_vector": "Network",
                "cvss_authentication": "None required",
                "cvss_availability_impact": "Partial",
                "cvss_base": 5.0,
                "cvss_confidentiality_impact": "None",
                "cvss_exploit": 10.0,
                "cvss_impact": 2.9,
                "cvss_integrity_impact": "None",
                "cvss_vector": [
                    "AV:N",
                    "AC:L",
                    "Au:N",
                    "C:N",
                    "I:N",
                    "A:P"
                ],
                "cweid": "CWE-399",
                "mod_date": "26-08-2011",
                "pub_date": "07-11-2006",
                "summary": "OpenLDAP before 2.3.29 allows remote attackers to cause a denial of service (daemon crash) via LDAP BIND requests with long authcid names, which triggers an assertion failure."
            }
        },
        {
            "CVE-2006-6493": {
                "cveid": "CVE-2006-6493",
                "cvss_access_complexity": "High",
                "cvss_access_vector": "Network",
                "cvss_authentication": "None required",
                "cvss_availability_impact": "Partial",
                "cvss_base": 5.1,
                "cvss_confidentiality_impact": "Partial",
                "cvss_exploit": 4.9,
                "cvss_impact": 6.4,
                "cvss_integrity_impact": "Partial",
                "cvss_vector": [
                    "AV:N",
                    "AC:H",
                    "Au:N",
                    "C:P",
                    "I:P",
                    "A:P"
                ],
                "cweid": "CWE-0",
                "mod_date": "07-03-2011",
                "pub_date": "12-12-2006",
                "summary": "Buffer overflow in the krbv4_ldap_auth function in servers/slapd/kerberos.c in OpenLDAP 2.4.3 and earlier, when OpenLDAP is compiled with the --enable-kbind (Kerberos KBIND) option, allows remote attackers to execute arbitrary code via an LDAP bind request using the LDAP_AUTH_KRBV41 authentication method and long credential data."
            }
        },
        {
            "CVE-2007-5707": {
                "cveid": "CVE-2007-5707",
                "cvss_access_complexity": "Medium",
                "cvss_access_vector": "Network",
                "cvss_authentication": "None required",
                "cvss_availability_impact": "Complete",
                "cvss_base": 7.1,
                "cvss_confidentiality_impact": "None",
                "cvss_exploit": 8.6,
                "cvss_impact": 6.9,
                "cvss_integrity_impact": "None",
                "cvss_vector": [
                    "AV:N",
                    "AC:M",
                    "Au:N",
                    "C:N",
                    "I:N",
                    "A:C"
                ],
                "cweid": "CWE-399",
                "mod_date": "07-03-2011",
                "pub_date": "30-10-2007",
                "summary": "OpenLDAP before 2.3.39 allows remote attackers to cause a denial of service (slapd crash) via an LDAP request with a malformed objectClasses attribute.  NOTE: this has been reported as a double free, but the reports are inconsistent."
            }
        },
        {
            "CVE-2007-5708": {
                "cveid": "CVE-2007-5708",
                "cvss_access_complexity": "Medium",
                "cvss_access_vector": "Network",
                "cvss_authentication": "None required",
                "cvss_availability_impact": "Complete",
                "cvss_base": 7.1,
                "cvss_confidentiality_impact": "None",
                "cvss_exploit": 8.6,
                "cvss_impact": 6.9,
                "cvss_integrity_impact": "None",
                "cvss_vector": [
                    "AV:N",
                    "AC:M",
                    "Au:N",
                    "C:N",
                    "I:N",
                    "A:C"
                ],
                "cweid": "CWE-399",
                "mod_date": "07-03-2011",
                "pub_date": "30-10-2007",
                "summary": "slapo-pcache (overlays/pcache.c) in slapd in OpenLDAP before 2.3.39, when running as a proxy-caching server, allocates memory using a malloc variant instead of calloc, which prevents an array from being initialized properly and might allow attackers to cause a denial of service (segmentation fault) via unknown vectors that prevent the array from being null terminated."
            }
        },
        {
            "CVE-2011-4079": {
                "cveid": "CVE-2011-4079",
                "cvss_access_complexity": "Low",
                "cvss_access_vector": "Network",
                "cvss_authentication": "Requires single instance",
                "cvss_availability_impact": "Partial",
                "cvss_base": 4.0,
                "cvss_confidentiality_impact": "None",
                "cvss_exploit": 8.0,
                "cvss_impact": 2.9,
                "cvss_integrity_impact": "None",
                "cvss_vector": [
                    "AV:N",
                    "AC:L",
                    "Au:S",
                    "C:N",
                    "I:N",
                    "A:P"
                ],
                "cweid": "CWE-189",
                "mod_date": "06-01-2017",
                "pub_date": "27-10-2011",
                "summary": "Off-by-one error in the UTF8StringNormalize function in OpenLDAP 2.4.26 and earlier allows remote attackers to cause a denial of service (slapd crash) via a zero-length string that triggers a heap-based buffer overflow, as demonstrated using an empty postalAddressAttribute value in an LDIF entry."
            }
        },
        {
            "BID-83610": {
                "bugtraq_id": 83610,
                "class": "Failure to Handle Exceptional Conditions",
                "cve": [
                    "CVE-2006-6493"
                ],
                "local": "no",
                "remote": "yes",
                "title": "OpenLDAP CVE-2006-6493 Remote Security Vulnerability"
            }
        },
        {
            "BID-83843": {
                "bugtraq_id": 83843,
                "class": "Failure to Handle Exceptional Conditions",
                "cve": [
                    "CVE-2006-2754"
                ],
                "local": "no",
                "remote": "yes",
                "title": "OpenLDAP CVE-2006-2754 Remote Security Vulnerability"
            }
        }
    ]

For getting all information about a specific CVE, you must run the next command:

    python3 dagda.py vuln --cve_info CVE-2009-2890

The expected output for the previous query is shown below:

    [
        {
            "cveid": "CVE-2009-2890",
            "cvss_access_complexity": "Medium",
            "cvss_access_vector": "Network",
            "cvss_authentication": "None required",
            "cvss_availability_impact": "None",
            "cvss_base": 4.3,
            "cvss_confidentiality_impact": "None",
            "cvss_exploit": 8.6,
            "cvss_impact": 2.9,
            "cvss_integrity_impact": "Partial",
            "cvss_vector": [
                "AV:N",
                "AC:M",
                "Au:N",
                "C:N",
                "I:P",
                "A:N"
            ],
            "cweid": "CWE-79",
            "mod_date": "20-08-2009",
            "pub_date": "20-08-2009",
            "summary": "Cross-site scripting (XSS) vulnerability in results.php in PHP Scripts Now Riddles allows remote attackers to inject arbitrary web script or HTML via the searchquery parameter."
        }
    ]

If you want to know more details about dagda.py vuln, type python3 dagda.py vuln --help or see the vuln sub-command in the wiki page.

Database contents

The database is called vuln_database and there are 10 collections:

Analyzing docker images/containers

In the next subsections, both, performing static analysis of known vulnerabilities, trojans, viruses, malware & other malicious threats and monitoring running docker containers for detecting anomalous activities will be described in depth.

Performing static analysis of known vulnerabilities and other malicious threats

One of the main Dagda targets is perform the analysis of known vulnerabilities, trojans, viruses, malware & other malicious threats in docker images/containers, so if you want perform an analysis over a docker image/container, you must type:

    python3 dagda.py check --docker_image jboss/wildfly

See the check sub-command wiki page for details.

The expected output for the previous command will be the next one. In this output, Dagda responses with the analysis id.

    {
        "id": "58667994ed253915723c50e7",
        "msg": "Accepted the analysis of <jboss/wildfly>"
    }

Also, if you want run a static analysis in a remote way, you can use the agent sub-command:

    python3 dagda.py agent localhost:5000 -i jboss/wildfly

The expected output for the previous command will be the next one. In this output, Dagda responses with the analysis id.

    {
        "id": "58667994ed253915723c50e7",
        "image_name": "jboss/wildfly"
    }

If you want review a concrete docker analysis, you must type:

    python3 dagda.py history <DOCKER_IMAGE_NAME_HERE> --id <REPORT_ID_HERE>

For more details about dagda.py history, type python3 dagda.py history --help or see the history sub-command in the wiki page.

The analysis can take several minutes for finishing, so be patient. If you typed the previous command, when you type python3 dagda.py history jboss/wildfly --id 58667994ed253915723c50e7, the expected output looks like as shown below.

    {
        "id": "58667994ed253915723c50e7",
        "image_name": "jboss/wildfly",
        "status": "Completed",
        "timestamp": "2016-12-14 13:17:12.802486",
        "static_analysis": {
            "malware_binaries": [
                {
                    "file": "/tmp/test/removal-tool.exe",
                    "malware": "Worm.Sober"
                },
                {
                    "file": "/tmp/test/error.hta",
                    "malware": "VBS.Inor.D"
                }
            ],
            "os_packages": {
                "total_os_packages": 182,
                "vuln_os_packages": 41,
                "ok_os_packages": 141,
                "os_packages_details": [
                    {
                        "product": "sed",
                        "version": "4.2.2",
                        "is_vulnerable": false,
                        "is_false_positive": false,
                        "vulnerabilities": []
                    },
                    {
                        "product": "grep",
                        "version": "2.20",
                        "is_vulnerable": true,
                        "is_false_positive": false,
                        "vulnerabilities": [
                            {
                                "CVE-2015-1345": {
                                    "cveid": "CVE-2015-1345",
                                    "cvss_access_complexity": "Low",
                                    "cvss_access_vector": "Local access",
                                    "cvss_authentication": "None required",
                                    "cvss_availability_impact": "Partial",
                                    "cvss_base": 2.1,
                                    "cvss_confidentiality_impact": "None",
                                    "cvss_exploit": 3.9,
                                    "cvss_impact": 2.9,
                                    "cvss_integrity_impact": "None",
                                    "cvss_vector": [
                                        "AV:L",
                                        "AC:L",
                                        "Au:N",
                                        "C:N",
                                        "I:N",
                                        "A:P"
                                    ],
                                    "cweid": "CWE-119",
                                    "mod_date": "23-12-2016",
                                    "pub_date": "12-02-2015",
                                    "summary": "The bmexec_trans function in kwset.c in grep 2.19 through 2.21 allows local users to cause a denial of service (out-of-bounds heap read and crash) via crafted input when using the -F option."
                                }
                            }
                        ]
                    },
                    {
                        "product": "lua",
                        "version": "5.1.4",
                        "is_vulnerable": true,
                        "is_false_positive": false,
                        "vulnerabilities": [
                            {
                                "CVE-2014-5461": {
                                    "cveid": "CVE-2014-5461",
                                    "cvss_access_complexity": "Low",
                                    "cvss_access_vector": "Network",
                                    "cvss_authentication": "None required",
                                    "cvss_availability_impact": "Partial",
                                    "cvss_base": 5.0,
                                    "cvss_confidentiality_impact": "None",
                                    "cvss_exploit": 10.0,
                                    "cvss_impact": 2.9,
                                    "cvss_integrity_impact": "None",
                                    "cvss_vector": [
                                        "AV:N",
                                        "AC:L",
                                        "Au:N",
                                        "C:N",
                                        "I:N",
                                        "A:P"
                                    ],
                                    "cweid": "CWE-119",
                                    "mod_date": "06-01-2017",
                                    "pub_date": "04-09-2014",
                                    "summary": "Buffer overflow in the vararg functions in ldo.c in Lua 5.1 through 5.2.x before 5.2.3 allows context-dependent attackers to cause a denial of service (crash) via a small number of arguments to a function with a large number of fixed arguments."
                                }
                            },
                            {
                                "BID-34237": {
                                    "bugtraq_id": 34237,
                                    "class": "Unknown",
                                    "cve": [],
                                    "local": "no",
                                    "remote": "yes",
                                    "title": "Lua Unspecified Bytecode Verifier Security Vulnerability"
                                }
                            }
                        ]
                    },
                    [...]
                    , {
                        "product": "sqlite",
                        "version": "3.7.17",
                        "is_vulnerable": false,
                        "is_false_positive": false,
                        "vulnerabilities": []
                    }
                ]
            },
            "prog_lang_dependencies": {
                "vuln_dependencies": 9,
                "dependencies_details": {
                    "java": [
                        {
                            "product": "xalan-java",
                            "version": "2.5.2",
                            "product_file_path": "/opt/jboss/java/xalan.2.5.2.jar",
                            "is_vulnerable": true,
                            "is_false_positive": false,
                            "vulnerabilities": [
                                {
                                    "CVE-2014-0107": {
                                        "cveid": "CVE-2014-0107",
                                        "cvss_access_complexity": "Low",
                                        "cvss_access_vector": "Network",
                                        "cvss_authentication": "None required",
                                        "cvss_availability_impact": "Partial",
                                        "cvss_base": 7.5,
                                        "cvss_confidentiality_impact": "Partial",
                                        "cvss_exploit": 10.0,
                                        "cvss_impact": 6.4,
                                        "cvss_integrity_impact": "Partial",
                                        "cvss_vector": [
                                            "AV:N",
                                            "AC:L",
                                            "Au:N",
                                            "C:P",
                                            "I:P",
                                            "A:P"
                                        ],
                                        "cweid": "CWE-264",
                                        "mod_date": "06-01-2017",
                                        "pub_date": "15-04-2014",
                                        "summary": "The TransformerFactory in Apache Xalan-Java before 2.7.2 does not properly restrict access to certain properties when FEATURE_SECURE_PROCESSING is enabled, which allows remote attackers to bypass expected restrictions and load arbitrary classes or access external resources via a crafted (1) xalan:content-header, (2) xalan:entities, (3) xslt:content-header, or (4) xslt:entities property, or a Java property that is bound to the XSLT 1.0 system-property function."
                                    }
                                },
                                {
                                    "BID-66397": {
                                        "bugtraq_id": 66397,
                                        "class": "Input Validation Error",
                                        "cve": [
                                            "CVE-2014-0107"
                                        ],
                                        "local": "no",
                                        "remote": "yes",
                                        "title": "Apache Xalan-Java Library CVE-2014-0107 Security Bypass Vulnerability"
                                    }
                                }
                            ]
                        },
                        {
                            "product": "jboss_wildfly_application_server",
                            "version": "-",
                            "product_file_path": "/opt/jboss/java/jboss_wildfly_application_server.jar",
                            "is_vulnerable": true,
                            "is_false_positive": false,
                            "vulnerabilities": [
                                {
                                    "CVE-2014-0018": {
                                        "cveid": "CVE-2014-0018",
                                        "cvss_access_complexity": "Medium",
                                        "cvss_access_vector": "Local access",
                                        "cvss_authentication": "None required",
                                        "cvss_availability_impact": "None",
                                        "cvss_base": 1.9,
                                        "cvss_confidentiality_impact": "None",
                                        "cvss_exploit": 3.4,
                                        "cvss_impact": 2.9,
                                        "cvss_integrity_impact": "Partial",
                                        "cvss_vector": [
                                            "AV:L",
                                            "AC:M",
                                            "Au:N",
                                            "C:N",
                                            "I:P",
                                            "A:N"
                                        ],
                                        "cweid": "CWE-264",
                                        "mod_date": "06-01-2017",
                                        "pub_date": "14-02-2014",
                                        "summary": "Red Hat JBoss Enterprise Application Platform (JBEAP) 6.2.0 and JBoss WildFly Application Server, when run under a security manager, do not properly restrict access to the Modular Service Container (MSC) service registry, which allows local users to modify the server via a crafted deployment."
                                    }
                                }
                            ]
                        },
                        [...]
                        , {
                            "product": "jboss_weld",
                            "version": "3.0.0",
                            "product_file_path": "/opt/jboss/java/jboss_weld.3.0.0.jar",
                            "is_vulnerable": true,
                            "is_false_positive": false,
                            "vulnerabilities": [
                                {
                                    "CVE-2014-8122": {
                                        "cveid": "CVE-2014-8122",
                                        "cvss_access_complexity": "Medium",
                                        "cvss_access_vector": "Network",
                                        "cvss_authentication": "None required",
                                        "cvss_availability_impact": "None",
                                        "cvss_base": 4.3,
                                        "cvss_confidentiality_impact": "Partial",
                                        "cvss_exploit": 8.6,
                                        "cvss_impact": 2.9,
                                        "cvss_integrity_impact": "None",
                                        "cvss_vector": [
                                            "AV:N",
                                            "AC:M",
                                            "Au:N",
                                            "C:P",
                                            "I:N",
                                            "A:N"
                                        ],
                                        "cweid": "CWE-362",
                                        "mod_date": "13-05-2015",
                                        "pub_date": "13-02-2015",
                                        "summary": "Race condition in JBoss Weld before 2.2.8 and 3.x before 3.0.0 Alpha3 allows remote attackers to obtain information from a previous conversation via vectors related to a stale thread state."
                                    }
                                }
                            ]
                        }
                    ],
                    "js": [],
                    "nodejs": [],
                    "php": [],
                    "python": [
                        {
                            "product": "lxml",
                            "version": "1.0.1",
                            "product_file_path": "/opt/jboss/python/lxml.1.0.1.py",
                            "is_vulnerable": true,
                            "is_false_positive": false,
                            "vulnerabilities": [
                                {
                                    "CVE-2014-3146": {
                                        "cveid": "CVE-2014-3146",
                                        "cvss_access_complexity": "Medium",
                                        "cvss_access_vector": "Network",
                                        "cvss_authentication": "None required",
                                        "cvss_availability_impact": "None",
                                        "cvss_base": 4.3,
                                        "cvss_confidentiality_impact": "None",
                                        "cvss_exploit": 8.6,
                                        "cvss_impact": 2.9,
                                        "cvss_integrity_impact": "Partial",
                                        "cvss_vector": [
                                            "AV:N",
                                            "AC:M",
                                            "Au:N",
                                            "C:N",
                                            "I:P",
                                            "A:N"
                                        ],
                                        "cweid": "CWE-0",
                                        "mod_date": "14-04-2015",
                                        "pub_date": "14-05-2014",
                                        "summary": "Incomplete blacklist vulnerability in the lxml.html.clean module in lxml before 3.3.5 allows remote attackers to conduct cross-site scripting (XSS) attacks via control characters in the link scheme to the clean_html function."
                                    }
                                }
                            ]
                        }
                    ],
                    "ruby": []
                }
            }
        }
    }

Monitoring running containers for detecting anomalous activities

Another of the main Dagda targets is perform the monitoring of running docker containers for detecting anomalous activities, so if you want perform the monitoring over a running docker container, you must type:

    python3 dagda.py monitor 69dbf26ab368 --start

See the monitor sub-command wiki page for details.

The expected output looks like as shown below:

    {
        "id": "586f7631ed25396a829baaf4",
        "image_name": "jboss/wildfly",
        "msg": "Monitoring of docker container with id <69dbf26ab368> started"
    }

You can stop the monitoring when you want if you type:

    python3 dagda.py monitor 69dbf26ab368 --stop

The expected output when you stop the monitoring over a running container looks like as shown below:

  {
      "id": "586f7631ed25396a829baaf4",
      "image_name": "jboss/wildfly",
      "timestamp": "2017-01-06 10:49:21.212508",
      "status": "Completed",
      "runtime_analysis": {
          "container_id": "69dbf26ab368",
          "start_timestamp": "2017-01-06 10:49:21.212508",
          "stop_timestamp": "2017-01-06 10:50:16.343847",
          "anomalous_activities_detected": {
              "anomalous_counts_by_severity": {
                  "Warning": 2
              },
              "anomalous_activities_details": [{
                  "output": "10:49:47.492517329: Warning Unexpected setuid call by non-sudo, non-root program (user=<NA> command=ping 8.8.8.8 uid=<NA>) container=thirsty_spence (id=69dbf26ab368)",
                  "priority": "Warning",
                  "rule": "Non sudo setuid",
                  "time": "2017-01-06 10:49:47.492516"
              }, {
                  "output": "10:49:53.181654702: Warning Unexpected setuid call by non-sudo, non-root program (user=<NA> command=ping 8.8.4.4 uid=<NA>) container=thirsty_spence (id=69dbf26ab368)",
                  "priority": "Warning",
                  "rule": "Non sudo setuid",
                  "time": "2017-01-06 10:49:53.181653"
              }]
          }
      }
  }

If you want review all your reports, see the history command.

Getting docker daemon events

Dagda includes the gathering of real time events from docker daemon, so if you want get all docker daemon events, you must type:

    python3 dagda.py docker events

The expected output looks like as shown below:

    [
        {
            "Action": "attach",
                "Actor": {
                    "Attributes": {
                        "build-date": "20171128",
                        "image": "jboss/wildfly",
                        "license": "GPLv2",
                        "name": "amazing_wilson",
                        "vendor": "CentOS"
                    },
                    "ID": "73c5ed015df661ce799baa685a39c32125a47b71f3476e9d452adc381fb8114c"
                },
                "Type": "container",
                "from": "jboss/wildfly",
                "id": "73c5ed015df661ce799baa685a39c32125a47b71f3476e9d452adc381fb8114c",
                "scope": "local",
                "status": "attach",
                "time": 1517323482,
                "timeNano": 1517323482957358115
            },
            {
                "Action": "create",
                "Actor": {
                    "Attributes": {
                        "build-date": "20171128",
                        "image": "jboss/wildfly",
                        "license": "GPLv2",
                        "name": "amazing_wilson",
                        "vendor": "CentOS"
                    },
                    "ID": "73c5ed015df661ce799baa685a39c32125a47b71f3476e9d452adc381fb8114c"
                },
                "Type": "container",
                "from": "jboss/wildfly",
                "id": "73c5ed015df661ce799baa685a39c32125a47b71f3476e9d452adc381fb8114c",
                "scope": "local",
                "status": "create",
                "time": 1517323482,
                "timeNano": 1517323482944595092
        }
    ]

If you want review all allowed filters for this command, see the docker command.

Bonus Track: Quick Start with Docker

This section describes the installation of Dagda using Docker containers, including the Mongo database and a container for Dagda, using docker-compose. The docker socket is shared with the Dagda container, so it is possible to check docker images and containers from the host where docker-compose is executed.

Execute the following commands in the root folder of Dagda and then, the Dagda server will start listening at port 5000:

    docker-compose build
    docker-compose up -d

Internal workflows

Below, a 10,000 foot diagram about the Dagda internal workflows is shown:

Dagda_internal_workflows

Troubleshooting

Typically, Dagda works fine, but some scenarios can cause problems. If you get some issue, check the Troubleshooting page for fixing it.

Change Log

See the Change Log page for details.

Bugs and Feedback

For bugs, questions and discussions please use the Github Issues or ping me on Twitter (@3grander).

dagda's People

Contributors

dependabot[bot] avatar eliasgranderubio avatar fossabot avatar jaguasch avatar jcortejoso avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dagda's Issues

How to scan a local image using dagda.

Short description

I was trying to do get an analysis of local image.

Reproduction steps

  1. I downloaded the vulnerabilities by 'python3 dagda.py vuln --init'
    and it showed the status updated after a while.
  2. 'python3 dagda.py check --docker_image nmap-dist:latest'
  3. 'python3 dagda.py history nmap-dist:latest --id 5ce791385ea27b39940ecfeb'

Expected results

I expected that by analyzing the docker image 'nmap-dist' and showing the report by using the command 'python3 dagda.py history nmap-dist:latest --id 5ce791385ea27b39940ecfeb' it would give a detailed analysis.

Actual results

{
    "err": 404,
    "msg": "History not found"
}

On which platforms did you notice this:

Please complete the following information:

ubuntu:18.04
OS Version: Linux esypc5 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:46:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Python version: [Python 3.6.7]
Docker version 18.09.6, build 481bc77
MongoDB version
db version v3.6.3
git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
OpenSSL version: OpenSSL 1.1.0g 2 Nov 2017
allocator: tcmalloc
modules: none
build environment:
distarch: x86_64
target_arch: x86_64

Backtrace

Please copy paste the backtrace here if available

Solution

Kindly help me with the same.

Dadga shows errors while scanning an untagged image

Hi @eliasgranderubio ,

Analyzing an image (testmonitor:1.0.0.test):

# docker images
REPOSITORY                                       TAG                 IMAGE ID            CREATED             SIZE                                                                  
docker.io/mongo                                  latest              8b2eaf28bb84        4 days ago          359.8 MB                                                              
testmonitor   					    1.0.0.test        211f3cc7e563        5 days ago           14.35 MB                                                              
docker.io/sysdig/falco                          latest              de4c13689a8d       2 weeks ago         379 MB     

If I launch the scan without the tag (1.0.0.test):

# python3 dagda.py check --docker_image testmonitor

The analysis starts, but dadga show errors in the server output:

Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/docker/api/client.py", line 214, in _raise_for_status
    response.raise_for_status()
  File "/usr/lib/python3.5/site-packages/requests/models.py", line 862, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localunixsocket/v1.24/containers/create

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dagda.py", line 130, in <module>
    main(DagdaCLIParser())
  File "dagda.py", line 63, in main
    ds.run()
  File "/home/scumfrog/Descargas/dagda/dagda/api/dagda_server.py", line 76, in run
    self._check_docker_by_image_name(item)
  File "/home/scumfrog/Descargas/dagda/dagda/api/dagda_server.py", line 140, in _check_docker_by_image_name
    evaluated_docker_image = analyzer.evaluate_image(item['image_name'], None)
  File "/home/scumfrog/Descargas/dagda/dagda/analysis/analyzer.py", line 47, in evaluate_image
    os_packages = os_info_extractor.get_soft_from_docker_image(self.dockerDriver, image_name)
  File "/home/scumfrog/Descargas/dagda/dagda/analysis/static/os/os_info_extractor.py", line 28, in get_soft_from_docker_image
    container_id = docker_driver.create_container(image_name, entrypoint='sleep 30')
  File "/home/scumfrog/Descargas/dagda/dagda/driver/docker_driver.py", line 77, in create_container
    host_config=host_config)
  File "/usr/lib/python3.5/site-packages/docker/api/container.py", line 446, in create_container
    return self.create_container_from_config(config, name)
  File "/usr/lib/python3.5/site-packages/docker/api/container.py", line 457, in create_container_from_config
    return self._result(res, True)
  File "/usr/lib/python3.5/site-packages/docker/api/client.py", line 220, in _result
    self._raise_for_status(response)
  File "/usr/lib/python3.5/site-packages/docker/api/client.py", line 216, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/lib/python3.5/site-packages/docker/errors.py", line 30, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found for url: http+docker://localunixsocket/v1.24/containers/create ("No such image: testmonitor:latest")

As you can see, automatically uses the "latest" tag, but the scan status is "analyzing". Should not end as "failed"?

    {
        "anomalies": 0,
        "image_name": "testmonitor",
        "libs_vulns": 0,
        "os_vulns": 0,
        "reportid": "58f5c6a508540e553a0bf356",
        "start_date": "2017-04-18 07:56:21.563832",
        "status": "Analyzing"
    }

Instead if you pass the tag, it works correctly without giving the last error in the server output:

    {
        "anomalies": 0,
        "image_name": "testmonitor:1.0.0.test",
        "libs_vulns": 0,
        "os_vulns": 0,
        "reportid": "58f5c6c008540e553a0bf357",
        "start_date": "2017-04-18 07:56:48.285536",
        "status": "Analyzing"
    },

Error

Hola Elias como vas.
Estoy probando el software Dagda, y me surge el siguiente error.

1- clone repositorio, e instale requerimientos
2. luego en dagda.py
Me da el siguiente error.

Traceback (most recent call last):
File "dagda.py", line 106, in
main(DagdaCLIParser())
File "dagda.py", line 102, in main
print(json.dumps(json.loads(r.content.decode('utf-8')), sort_keys=True, indent=4))
File "/usr/lib/python3.5/json/init.py", line 319, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.5/json/decoder.py", line 342, in decode
raise JSONDecodeError("Extra data", s, end)

Procedo a entrar al dagda.py
comento lo siguiente.

if r is not None:
print(json.dumps(json.loads(r.content.decode('utf-8')), sort_keys=True, indent=4))

Y cuando ejecuto.

python3 dagda.py history namedocker --id SCANID

Por su puesto no me muestra nada, pero corre sin problema. es decir el problema esta cuando vota el resultado del análisis. que crees que podría ser.

vuln first sync error

Hello,

Trying to do my first sync of the Vuln DB. it fails with this message:

dagda     |  <2017-04-28 12:38:07,069> <ERROR> <DagdaServer> <dagda_server> <Unexpected exception of type JoblibConnectionError occured: ('Multiprocessing exception:
...........................................................................
/opt/app/dagda.py in <module>()
    130         if r is not None:
    131             print(json.dumps(json.loads(r.content.decode(\'utf-8\')), sort_keys=True, indent=4))
    132
    133 
    134 if __name__ == "__main__":
--> 135     main(DagdaCLIParser())
    136 
    137 
    138 
    139 

...........................................................................
/opt/app/dagda.py in main(parsed_args=<cli.command.start_cli_parser.StartCLIParser object>)
     58                          mongodb_port=parsed_args.get_mongodb_port(),
     59                          mongodb_ssl=parsed_args.is_mongodb_ssl_enabled(),
     60                          mongodb_user=parsed_args.get_mongodb_user(),
     61                          mongodb_pass=parsed_args.get_mongodb_pass(),
     62                          falco_rules_filename=parsed_args.get_falco_rules_filename())
---> 63         ds.run()
        ds.run = <bound method DagdaServer.run of <api.dagda_server.DagdaServer object>>
     64 
     65     else:
     66         dagda_base_url = get_dagda_base_url()
     67         # -- Executes vuln sub-command

...........................................................................
/opt/app/api/dagda_server.py in run(self=<api.dagda_server.DagdaServer object>)
     69         if edn_pid == 0:
     70             try:
     71                 while True:
     72                     item = InternalServer.get_dagda_edn().get()
     73                     if item[\'msg\'] == \'init_db\':
---> 74                         self._init_or_update_db()
        self._init_or_update_db = <function DagdaServer._init_or_update_db>
     75                     elif item[\'msg\'] == \'check_image\':
     76                         self._check_docker_by_image_name(item)
     77                     elif item[\'msg\'] == \'check_container\':
     78                         self._check_docker_by_container_id(item)

...........................................................................
/opt/app/api/dagda_server.py in _init_or_update_db()
    121         try:
    122             InternalServer.get_mongodb_driver().insert_init_db_process_status(
    123                 {\'status\': \'Initializing\', \'timestamp\': datetime.datetime.now().timestamp()})
    124             # Init db
    125             db_composer = DBComposer()
--> 126             db_composer.compose_vuln_db()
        db_composer.compose_vuln_db = <bound method DBComposer.compose_vuln_db of <vulnDB.db_composer.DBComposer object>>
    127             InternalServer.get_mongodb_driver().insert_init_db_process_status(
    128                 {\'status\': \'Updated\', \'timestamp\': datetime.datetime.now().timestamp()})
    129         except Exception as ex:
    130             message = "Unexpected exception of type {0} occured: {1!r}".format(type(ex).__name__,  ex.args)

...........................................................................
/opt/app/vulnDB/db_composer.py in compose_vuln_db(self=<vulnDB.db_composer.DBComposer object>)
     92                 self.mongoDbDriver.bulk_insert_bids(bid_items_list)
     93                 bid_items_list.clear()
     94             # Set the new max bid
     95             max_bid = 94417
     96         # Updating BugTraqs from http://www.securityfocus.com/
---> 97         bid_items_array = get_bug_traqs_lists_from_online_mode(bid_downloader(first_bid=max_bid+1, last_bid=97200))
        bid_items_array = [[], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], ...]
        max_bid = 94417
     98         for bid_items_list in bid_items_array:
     99             self.mongoDbDriver.bulk_insert_bids(bid_items_list)
    100             bid_items_list.clear()
    101 

...........................................................................
/opt/app/vulnDB/bid_downloader.py in bid_downloader(first_bid=94418, last_bid=97200)
     72             return json.dumps(prepare_output(title, bugtraq_id, vuln_products), sort_keys=True)
     73 
     74 
     75 # Executes the main function called get_bid in a parallel way
     76 def bid_downloader(first_bid, last_bid):
---> 77     output_list = Parallel(n_jobs=100)(delayed(get_bid)(i) for i in range(first_bid, last_bid + 1))
        output_list = undefined
        first_bid = 94418
        last_bid = 97200
     78     return [x for x in output_list if x is not None]
     79 
     80 
     81 

...........................................................................
/usr/local/lib/python3.4/site-packages/joblib/parallel.py in __call__(self=Parallel(n_jobs=100), iterable=<generator object <genexpr>>)
    763             if pre_dispatch == "all" or n_jobs == 1:
    764                 # The iterable was consumed all at once by the above for loop.
    765                 # No need to wait for async callbacks to trigger to
    766                 # consumption.
    767                 self._iterating = False
--> 768             self.retrieve()
        self.retrieve = <bound method Parallel.retrieve of Parallel(n_jobs=100)>
    769             # Make sure that we get a last message telling us we are done
    770             elapsed_time = time.time() - self._start_time
    771             self._print(\'Done %3i out of %3i | elapsed: %s finished\',
    772                         (len(self._output), len(self._output),

---------------------------------------------------------------------------
Sub-process traceback:
---------------------------------------------------------------------------
ConnectionError                                    Fri Apr 28 12:38:06 2017
PID: 43                                 Python 3.4.5: /usr/local/bin/python\n...........................................................................
/usr/local/lib/python3.4/site-packages/joblib/parallel.py in __call__(self=<joblib.parallel.BatchedCalls object>)
    126     def __init__(self, iterator_slice):
    127         self.items = list(iterator_slice)
    128         self._size = len(self.items)
    129
    130     def __call__(self):
--> 131         return [func(*args, **kwargs) for func, args, kwargs in self.items]
        self.items = [(<function get_bid>, (94999,), {})]
    132
    133     def __len__(self):
    134         return self._size
    135

...........................................................................
/usr/local/lib/python3.4/site-packages/joblib/parallel.py in <listcomp>(.0=<list_iterator object>)
    126     def __init__(self, iterator_slice):
    127         self.items = list(iterator_slice)
    128         self._size = len(self.items)
    129
    130     def __call__(self):
--> 131         return [func(*args, **kwargs) for func, args, kwargs in self.items]\n        func = <function get_bid>
        args = (94999,)
        kwargs = {}
    132 
    133     def __len__(self):
    134         return self._size
    135 

...........................................................................
/opt/app/vulnDB/bid_downloader.py in get_bid(bugtraq_id=94999)
     57     return data
     58 
     59
     60 # Requests the bid, parses the HTML and prints the BugTraq info
     61 def get_bid(bugtraq_id):
---> 62     r = requests.get("http://www.securityfocus.com/bid/" + str(bugtraq_id))
        r = undefined\n        bugtraq_id = 94999
     63     if r.status_code == 200:
     64         try:
     65             body = r.content.decode("utf-8")
     66             body = body[body.index(\'<div id="vulnerability">\'):body.index(\'<span class="label">Not Vulnerable:</span>\')]

...........................................................................
/usr/local/lib/python3.4/site-packages/requests/api.py in get(url=\'http://www.securityfocus.com/bid/94999\', params=None, **kwargs={\'allow_redirects\': True})
     65     :return: :class:`Response <Response>` object
     66     :rtype: requests.Response
     67     """
     68 
     69     kwargs.setdefault(\'allow_redirects\', True)
---> 70     return request(\'get\', url, params=params, **kwargs)
        url = \'http://www.securityfocus.com/bid/94999\'
        params = None\n        kwargs = {\'allow_redirects\': True}
     71 
     72 
     73 def options(url, **kwargs):
     74     """Sends a OPTIONS request.

...........................................................................
/usr/local/lib/python3.4/site-packages/requests/api.py in request(method=\'get\', url=\'http://www.securityfocus.com/bid/94999\', **kwargs={\'allow_redirects\': True, \'params\': None})
     51 
     52     # By using the \'with\' statement we are sure the session is closed, thus we
     53     # avoid leaving sockets open which can trigger a ResourceWarning in some
     54     # cases, and look like a memory leak in others.
     55     with sessions.Session() as session:
---> 56         return session.request(method=method, url=url, **kwargs)
        session.request = <bound method Session.request of <requests.sessions.Session object>>
        method = \'get\'
        url = \'http://www.securityfocus.com/bid/94999\'
        kwargs = {\'allow_redirects\': True, \'params\': None}
     57 
     58 
     59 def get(url, params=None, **kwargs):
     60     """Sends a GET request.

...........................................................................
/usr/local/lib/python3.4/site-packages/requests/sessions.py in request(self=<requests.sessions.Session object>, method=\'get\', url=\'http://www.securityfocus.com/bid/94999\', params=None, data=None, headers=None, cookies=None, files=None, auth=None, timeout=None, allow_redirects=True, proxies={}, hooks=None, stream=None, verify=None, cert=None, json=None)
    470         send_kwargs = {
    471             \'timeout\': timeout,
    472             \'allow_redirects\': allow_redirects,
    473         }
    474         send_kwargs.update(settings)
--> 475         resp = self.send(prep, **send_kwargs)
        resp = undefined
        self.send = <bound method Session.send of <requests.sessions.Session object>>
        prep = <PreparedRequest [GET]>
        send_kwargs = {\'allow_redirects\': True, \'cert\': None, \'proxies\': OrderedDict(), \'stream\': False, \'timeout\': None, \'verify\': True}
    476
    477         return resp
    478
    479     def get(self, url, **kwargs):

...........................................................................
/usr/local/lib/python3.4/site-packages/requests/sessions.py in send(self=<requests.sessions.Session object>, request=<PreparedRequest [GET]>, **kwargs={\'cert\': None, \'proxies\': OrderedDict(), \'stream\': False, \'timeout\': None, \'verify\': True})
    591 
    592         # Start time (approximately) of the request
    593         start = datetime.utcnow()
    594 
    595         # Send the request
--> 596         r = adapter.send(request, **kwargs)
        r = undefined
        adapter.send = <bound method HTTPAdapter.send of <requests.adapters.HTTPAdapter object>>
        request = <PreparedRequest [GET]>
        kwargs = {\'cert\': None, \'proxies\': OrderedDict(), \'stream\': False, \'timeout\': None, \'verify\': True}
    597 
    598         # Total elapsed time of the request (approximately)
    599         r.elapsed = datetime.utcnow() - start
    600 

...........................................................................
/usr/local/lib/python3.4/site-packages/requests/adapters.py in send(self=<requests.adapters.HTTPAdapter object>, request=<PreparedRequest [GET]>, stream=False, timeout=<requests.packages.urllib3.util.timeout.Timeout object>, verify=True, cert=None, proxies=OrderedDict())
    468                     # Then, reraise so that we can handle the actual exception.
    469                     low_conn.close()
    470                     raise
    471 
    472         except (ProtocolError, socket.error) as err:
--> 473             raise ConnectionError(err, request=request)
        err = undefined
        request = <PreparedRequest [GET]>
    474 
    475         except MaxRetryError as e:
    476             if isinstance(e.reason, ConnectTimeoutError):
    477                 # TODO: Remove this in 3.0.0: see #2811

ConnectionError: (\'Connection aborted.\', ConnectionResetError(104, \'Connection reset by peer\'))',)>

What can i do to fix this ?

Thanks for your help,

Dagda not detecting malware binaries

I was trying to test the malware detection of dagda from the CLI, and it doesn't seem to pick up anything.

Steps I took:

  1. Started dagda server

  2. Performed proper assignment of environment variables and initiation of vulnerability database.

  3. Built an image from a dockerfile with the line:

'RUN git clone https://www.github.com/ytisf/theZoo'
**The github repo is literally just full of malware binaries **

  1. Ran the dagda check for on the docker image

Results

The dagda history shows no malware binaries being scanned. Not a single one. Vulnerability scanning for CVE's seems to work fine though.

###Specs

  • OS: Ubuntu 18.04
  • Python version: Python 3.4]
  • Docker version: 18.09.7
  • MongoDB version: mongo:latest

MongoDB port to be changed to non-standard port.

Short description

I was trying to deploy the dokcer-compose option on our environment, unfortunately I cant use the normal mongodb port 27017 as its in use already.. So for example I am using a host port 27018 and container port 27017.

Would it work ? We tried with some options without much luck. Please advice.

unable to start the dagda

when i was running this command, python3 dagda.py start

getting below error,
ubuntu@ip-172-31-11-35:~/docker-security/dagda/dagda$ python3 dagda.py start
<2018-09-28 19:50:12,142> <docker_driver> <init:39>
<2018-09-28 19:50:12,174> main:46 <'NoneType' object has no attribute 'events'>
Traceback (most recent call last):
File "dagda.py", line 35, in main
r = execute_dagda_cmd(cmd=cmd, args=parsed_args)
File "/home/ubuntu/docker-security/dagda/dagda/cli/dagda_cli.py", line 43, in execute_dagda_cmd
ds.run()
File "/home/ubuntu/docker-security/dagda/dagda/api/dagda_server.py", line 92, in run
docker_daemon_events_monitor.run()
File "/home/ubuntu/docker-security/dagda/dagda/analysis/runtime/docker_events_monitor.py", line 45, in run
for event in self.docker_driver.docker_events():
File "/home/ubuntu/docker-security/dagda/dagda/driver/docker_driver.py", line 114, in docker_events
return self.cli.events()
AttributeError: 'NoneType' object has no attribute 'events'
<2018-09-28 19:50:12,199> <dagda_server> run:103
<2018-09-28 19:50:12,200> <dagda_server> run:104

i followed your github page and performed the all necessary actions.

uncaught exception in _untar_layers

Short description

I was trying to test ubuntu:16.04

Reproduction steps

  1. Start Dagda server with the debug parameter ( python3 dagda.py start -d )
  2. Type the Dagda CLI command python3 dagda.py check --docker_image ubuntu:16.04

Expected results

I expected that by analyzing the docker image ubuntu:16.04 and showing the report.

Actual results

Instead of a report, what I saw was that error Unexpected exception of type PermissionError occured: (1, 'Operation not permitted')

On which platforms did you notice this:

Please complete the following information:

  • OS: Ubuntu
  • OS Version: Linux PC 4.15.0-34-generic #37-Ubuntu SMP Mon Aug 27 15:21:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Python version: Python 3.6.5
  • Docker version Docker version 17.12.1-ce, build 7390fc6
  • MongoDB version 2.6.12

Backtrace

<2018-09-20 18:59:59,109> <ERROR> <DagdaServer> <analyzer> <evaluate_image:108> <Unexpected exception of type PermissionError occured: (1, 'Operation not permitted')>
Traceback (most recent call last):
  File "$HOME/dagda/dagda/analysis/analyzer.py", line 74, in evaluate_image
    image_name=image_name)
  File "$HOME/dagda/dagda/analysis/static/util/utils.py", line 47, in extract_filesystem_bundle
    _untar_layers(temporary_dir, layers)
  File "$HOME/dagda/dagda/analysis/static/util/utils.py", line 82, in _untar_layers
    tarfile.extract(output[member_name], path=dir, set_attrs=False)
  File "/usr/lib/python3.6/tarfile.py", line 2050, in extract
    numeric_owner=numeric_owner)
  File "/usr/lib/python3.6/tarfile.py", line 2126, in _extract_member
    self.makedev(tarinfo, targetpath)
  File "/usr/lib/python3.6/tarfile.py", line 2200, in makedev
    os.makedev(tarinfo.devmajor, tarinfo.devminor))
PermissionError: [Errno 1] Operation not permitted

also I make a little modyfication in dagda/analysis/static/util/utils.py

def _untar_layers(dir, layers):
    output = {}
    # Untar layer filesystem bundle
    for layer in layers:
        tarfile = TarFile(dir + "/" + layer)
        for member in tarfile.getmembers():
            output[member.name] = member
    for member_name in output:
        try:
            tarfile.extract(output[member_name], path=dir, set_attrs=False)
        except (ValueError, ReadError):
            pass
        except:
            DagdaLogger.get_logger().debug('tar custom: ' + member_name)          

    # Clean up
    for layer in layers:
        clean_up(dir + "/" + layer[:-10])

and here is the result of the debug function:

<2018-09-20 18:58:26,813> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/audio>
<2018-09-20 18:58:26,813> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/audio1>
<2018-09-20 18:58:26,813> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/audio2>
<2018-09-20 18:58:26,813> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/audio3>
<2018-09-20 18:58:26,813> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/audioctl>
<2018-09-20 18:58:26,813> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/console>
<2018-09-20 18:58:26,813> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/dsp>
<2018-09-20 18:58:26,813> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/dsp1>
<2018-09-20 18:58:26,813> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/dsp2>
<2018-09-20 18:58:26,814> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/dsp3>
<2018-09-20 18:58:26,814> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/full>
<2018-09-20 18:58:26,814> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/kmem>
<2018-09-20 18:58:26,814> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/loop0>
<2018-09-20 18:58:26,814> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/loop1>
<2018-09-20 18:58:26,814> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/loop2>
<2018-09-20 18:58:26,814> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/loop3>
<2018-09-20 18:58:26,814> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/loop4>
<2018-09-20 18:58:26,814> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/loop5>
<2018-09-20 18:58:26,814> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/loop6>
<2018-09-20 18:58:26,814> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/loop7>
<2018-09-20 18:58:26,814> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/mem>
<2018-09-20 18:58:26,814> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/midi0>
<2018-09-20 18:58:26,814> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/midi00>
<2018-09-20 18:58:26,814> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/midi01>
<2018-09-20 18:58:26,815> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/midi02>
<2018-09-20 18:58:26,815> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/midi03>
<2018-09-20 18:58:26,815> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/midi1>
<2018-09-20 18:58:26,815> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/midi2>
<2018-09-20 18:58:26,815> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/midi3>
<2018-09-20 18:58:26,815> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/mixer>
<2018-09-20 18:58:26,815> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/mixer1>
<2018-09-20 18:58:26,815> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/mixer2>
<2018-09-20 18:58:26,815> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/mixer3>
<2018-09-20 18:58:26,815> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/mpu401data>
<2018-09-20 18:58:26,815> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/mpu401stat>
<2018-09-20 18:58:26,815> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/null>
<2018-09-20 18:58:26,815> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/port>
<2018-09-20 18:58:26,816> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram0>
<2018-09-20 18:58:26,816> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram1>
<2018-09-20 18:58:26,816> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram10>
<2018-09-20 18:58:26,816> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram11>
<2018-09-20 18:58:26,816> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram12>
<2018-09-20 18:58:26,816> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram13>
<2018-09-20 18:58:26,816> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram14>
<2018-09-20 18:58:26,816> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram15>
<2018-09-20 18:58:26,816> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram16>
<2018-09-20 18:58:26,816> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram2>
<2018-09-20 18:58:26,817> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram3>
<2018-09-20 18:58:26,817> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram4>
<2018-09-20 18:58:26,817> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram5>
<2018-09-20 18:58:26,817> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram6>
<2018-09-20 18:58:26,817> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram7>
<2018-09-20 18:58:26,817> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram8>
<2018-09-20 18:58:26,817> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/ram9>
<2018-09-20 18:58:26,817> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/random>
<2018-09-20 18:58:26,817> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/rmidi0>
<2018-09-20 18:58:26,817> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/rmidi1>
<2018-09-20 18:58:26,817> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/rmidi2>
<2018-09-20 18:58:26,817> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/rmidi3>
<2018-09-20 18:58:26,817> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/sequencer>
<2018-09-20 18:58:26,817> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/smpte0>
<2018-09-20 18:58:26,818> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/smpte1>
<2018-09-20 18:58:26,818> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/smpte2>
<2018-09-20 18:58:26,818> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/smpte3>
<2018-09-20 18:58:26,818> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/sndstat>
<2018-09-20 18:58:26,818> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/tty>
<2018-09-20 18:58:26,818> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/tty0>
<2018-09-20 18:58:26,818> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/tty1>
<2018-09-20 18:58:26,818> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/tty2>
<2018-09-20 18:58:26,818> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/tty3>
<2018-09-20 18:58:26,818> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/tty4>
<2018-09-20 18:58:26,818> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/tty5>
<2018-09-20 18:58:26,818> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/tty6>
<2018-09-20 18:58:26,818> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/tty7>
<2018-09-20 18:58:26,818> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/tty8>
<2018-09-20 18:58:26,818> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/tty9>
<2018-09-20 18:58:26,819> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/urandom>
<2018-09-20 18:58:26,819> <DEBUG> <DagdaServer> <utils> <_untar_layers:86> <tar custom: dev/zero>

Solution

A possible solution for this issue would be modify the _untar_layers function to catch that kind of exception.

Dagda won't start, complaining about Docker Module not found

Short description

I was trying to install dagda on Ubuntu 18.04

Reproduction steps

Install all pre-reqs including docker and mongo
git clone the repo
try and start dagda with the following command:

sudo python3 dagda.py start

Expected results

I expected the dagda service to start

Actual results

Instead of the service starting, it gave me that there is no "docker" module. Docker is installed. I can run a docker pull hello-world docker run hello-world

On which platforms did you notice this:

Please complete the following information:

  • OS: Ubuntu
  • OS Version: 18.04
  • Python version: 3.7
  • Docker version: Docker version 18.09.1, build 4c52b90
  • MongoDB version 4.0.5

Backtrace

 <ERROR> <DagdaServer> <dagda> <main:46> <No module named 'docker'>
Traceback (most recent call last):
  File "dagda.py", line 35, in main
    r = execute_dagda_cmd(cmd=cmd, args=parsed_args)
  File "/usr/lib/dagda/dagda/cli/dagda_cli.py", line 32, in execute_dagda_cmd
    from api.dagda_server import DagdaServer
  File "/usr/lib/dagda/dagda/api/dagda_server.py", line 27, in <module>
    from api.internal.internal_server import InternalServer
  File "/usr/lib/dagda/dagda/api/internal/internal_server.py", line 22, in <module>
    from driver.docker_driver import DockerDriver
  File "/usr/lib/dagda/dagda/driver/docker_driver.py", line 20, in <module>
    import docker
ModuleNotFoundError: No module named 'docker'

failed to strat dagda

Short description

I was trying to strat dagda with this command "python3 dagda.py start" but nothing happen (no error to). If I aviod this step and try "python3 dagda.py vuln --init" I have a newConnectionError .

Reproduction steps

  1. Installation of Docker
  2. Installation of MongoDB
  3. Installation of kernel headers in the host OS
  4. sudo pip3 install -r requirements.txt
  5. Avoid Syslog as your Docker Logging Driver
  6. sudo service mongod | falco | docker start
  7. python3 dagda.py start
    => nothing happen after more than one hours
  8. CTRL+C
    => no error
  9. export DAGDA_HOST='127.0.0.1'
  10. export DAGDA_PORT=5000
  11. python3 dagda.py vuln --init
    => <HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /v1/vuln/init (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fdcc67a9e10>: Failed to establish a new connection: [Errno 111] Connection refused',))>
    root@debian:~/dagda-master/dagda#

On which platforms did you notice this:

  • OS: Debian
  • OS Version: 9.3
  • Python version: Python 3.5.3
  • Docker version: 17.12.0-ce
  • MongoDB version 3.6.2

TY for your help ! :)

JSON decoder ValueError in CLI with None value

I create this new issue for fixing the CLI problem described by @scumfrog in #10. The issue details is shown in this comment.

The error that he gets is the next one:

# python3 dagda.py vuln --init                        
Traceback (most recent call last):
  File "dagda.py", line 125, in <module>
    main(DagdaCLIParser())
  File "dagda.py", line 121, in main
    print(json.dumps(json.loads(r.content.decode('utf-8')), sort_keys=True, indent=4))
  File "/usr/lib64/python3.4/json/__init__.py", line 318, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.4/json/decoder.py", line 343, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.4/json/decoder.py", line 361, in raw_decode
    raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)

Unexpected exception of type ReadError occurred while untaring the docker image

Short description

Log produced in debug mode, when use check command to analyze a docker image.
I have tried to check different images, and the log produced the same every time.

Reproduction steps

  1. python3 dagda/dagda.py check --docker_image grafana/grafana

Actual results

.....
<2019-03-13 21:46:44,609> <DEBUG> <DagdaServer> <utils> <_untar_layers:87> <Unexpected exception of type ReadError occurred while untaring the docker image: ('unexpected end of data',)>
<2019-03-13 21:46:44,609> <DEBUG> <DagdaServer> <utils> <_untar_layers:87> <Unexpected exception of type ReadError occurred while untaring the docker image: ('unexpected end of data',)>
<2019-03-13 21:46:44,609> <DEBUG> <DagdaServer> <utils> <_untar_layers:87> <Unexpected exception of type ReadError occurred while untaring the docker image: ('unexpected end of data',)>
<2019-03-13 21:46:44,609> <DEBUG> <DagdaServer> <utils> <_untar_layers:87> <Unexpected exception of type ReadError occurred while untaring the docker image: ('unexpected end of data',)>
<2019-03-13 21:46:44,609> <DEBUG> <DagdaServer> <utils> <_untar_layers:87> <Unexpected exception of type ReadError occurred while untaring the docker image: ('unexpected end of data',)>
<2019-03-13 21:46:44,609> <DEBUG> <DagdaServer> <utils> <_untar_layers:87> <Unexpected exception of type ReadError occurred while untaring the docker image: ('unexpected end of data',)>
<2019-03-13 21:46:44,610> <DEBUG> <DagdaServer> <utils> <_untar_layers:87> <Unexpected exception of type ReadError occurred while untaring the docker image: ('unexpected end of data',)>
<2019-03-13 21:46:44,610> <DEBUG> <DagdaServer> <utils> <_untar_layers:87> <Unexpected exception of type ReadError occurred while untaring the docker image: ('unexpected end of data',)>
<2019-03-13 21:46:44,610> <DEBUG> <DagdaServer> <utils> <_untar_layers:87> <Unexpected exception of type ReadError occurred while untaring the docker image: ('unexpected end of data',)>
<2019-03-13 21:46:44,610> <DEBUG> <DagdaServer> <utils> <_untar_layers:87> <Unexpected exception of type ReadError occurred while untaring the docker image: ('unexpected end of data',)>
<2019-03-13 21:46:44,611> <DEBUG> <DagdaServer> <utils> <_untar_layers:87> <Unexpected exception of type ReadError occurred while untaring the docker image: ('unexpected end of data',)>
<2019-03-13 21:46:44,817> <DEBUG> <DagdaServer> <analyzer> <evaluate_image:83> <OS packages from the docker image retrieved>
<2019-03-13 21:46:44,818> <DEBUG> <DagdaServer> <analyzer> <_threaded_malware:226> <Retrieving malware files from the docker image ...>
<2019-03-13 21:46:44,818> <DEBUG> <DagdaServer> <analyzer> <_threaded_dependencies:239> <Retrieving dependencies from the docker image ...>
<2019-03-13 21:47:01,342> <DEBUG> <DagdaServer> <docker_driver> <docker_stop:110> <99d67d225edd3e6b972ce7b72680fdff438c696c23c66316ad93add1f0fa2587>
<2019-03-13 21:47:01,417> <DEBUG> <DagdaServer> <analyzer> <_threaded_malware:232> <Malware files from the docker image retrieved>
<2019-03-13 21:47:20,170> <DEBUG> <DagdaServer> <docker_driver> <docker_stop:110> <fb54130bc5df8a034ec958a353e9b8139e8d1905d80be3b73c56923b279f7d72>
<2019-03-13 21:47:20,254> <DEBUG> <DagdaServer> <analyzer> <_threaded_dependencies:246> <Dependencies from the docker image retrieved>
<2019-03-13 21:47:20,491> <DEBUG> <DagdaServer> <analyzer> <evaluate_image:113> <Preparing analysis output ...>
<2019-03-13 21:47:25,804> <DEBUG> <DagdaServer> <analyzer> <evaluate_image:123> <Analysis output completed>
<2019-03-13 21:47:25,804> <DEBUG> <DagdaServer> <analyzer> <evaluate_image:127> <EXIT from the method for analyzing a docker image>

Unexpected exception of type DagdaError occurred: 'Linux image distribution has not the "/etc/os-release" file.'

Short description

I was trying to do scan some images and obtain the error in the dagda server

Unexpected exception of type DagdaError occurred: 'Linux image distribution has not the "/etc/os-release" file.'

Reproduction steps

  1. Start Dagda environment from docker-compose.
  2. Init vuln database
  3. Type the Dagda CLI command python3 dagda.py check --docker_image
  4. View Dagda Server logs: docker-compose logs -f -t dagda

Expected results

I expected that by analyzing the docker image and obtain the report

On which platforms did you notice this:

  • OS: centos-release-7-6.1810.2.el7.centos.x86_64
  • OS Version: Linux cosdagda.odigeo.org 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Python version: Python 3.6.3
  • Docker version: Docker version 18.09.1, build 4c52b90
  • MongoDB version: default latest image from docker hub

Backtrace

dagda     | 2019-03-11T16:44:00.998883556Z <2019-03-11 16:44:00,997> <ERROR> <DagdaServer> <analyzer> <evaluate_image:102> <Unexpected exception of type DagdaError occurred: 'Linux image distribution has not the "/etc/os-release" file.'>

Add first detection date to the cves image report

Short description

I analyze a private registry (with more than 100 images) and I keep a history of the scans I do once a week. To be able to track the vulnerabilities, it would be very useful to have the date on which a vulnerable was identified (CVE for example).
In the section Expected results I add the JSON of the analysis of an image by adding the parameter "first_detection" : "20190312" inside the CVE.

Expected results

{
        "id": "58667994ed253915723c50e7",
        "image_name": "jboss/wildfly",
        "status": "Completed",
        "timestamp": "2016-12-14 13:17:12.802486",
        "static_analysis": {
            ...,...,
            "os_packages": {
                {.., ..}
                "os_packages_details": [
                    {..,..}
                    {
                        ...,
						...
                        "vulnerabilities": [
                            {
                                "CVE-2015-1345": {
				   "first_detection" : "20190312"
                                    "cveid": "CVE-2015-1345",
                                    "cvss_access_complexity": "Low",
                                    "cvss_access_vector": "Local access",
                                    "cvss_authentication": "None required",
                                    "cvss_availability_impact": "Partial",
                                    "cvss_base": 2.1,
                                    "cvss_confidentiality_impact": "None",
                                    "cvss_exploit": 3.9,
                                    "cvss_impact": 2.9,
                                    "cvss_integrity_impact": "None",
                                    "cvss_vector": [
                                        "AV:L",
                                        "AC:L",
                                        "Au:N",
                                        "C:N",
                                        "I:N",
                                        "A:P"
                                    ],
                                    "cweid": "CWE-119",
                                    "mod_date": "23-12-2016",
                                    "pub_date": "12-02-2015",
                                    "summary": "The bmexec_trans function in kwset.c in grep 2.19 through 2.21 allows local users to cause a denial of service (out-of-bounds heap read and crash) via crafted input when using the -F option."
                                }
                            }
                        ]
                    },

Adicional tests

Additionally we have tried to make a post as indicated in the wiki for adds a new image analysis to the image history. We thought we could add a new analysis and modify the json of the post to accept this new field. But when we tried to make the post with the data of the example without modifying we got an error:

Post like the example without modify

curl http://localhost:5000/v1/history/image:test -X POST -d {"image_name":"jboss/wildfly","status":"Completed","timestamp":1494609523.342605,"static_analysis":{"malware_binaries":[{"file":"/tmp/test/removal-tool.exe","malware":"Worm.Sober"},{"file":"/tmp/test/error.hta","malware":"VBS.Inor.D"}],"os_packages":{"total_os_packages":182,"vuln_os_packages":41,"ok_os_packages":141,"os_packages_details":[{"product":"sed","version":"4.2.2","is_vulnerable":false,"is_false_positive":false,"vulnerabilities":[]},{"product":"grep","version":"2.20","is_vulnerable":true,"is_false_positive":false,"vulnerabilities":[{"CVE-2015-1345":{"cveid":"CVE-2015-1345","cvss_access_complexity":"Low","cvss_access_vector":"Local access","cvss_authentication":"None required","cvss_availability_impact":"Partial","cvss_base":2.1,"cvss_confidentiality_impact":"None","cvss_exploit":3.9,"cvss_impact":2.9,"cvss_integrity_impact":"None","cvss_vector":["AV:L","AC:L","Au:N","C:N","I:N","A:P"],"cweid":"CWE-119","mod_date":"23-12-2016","pub_date":"12-02-2015","summary":"The bmexec_trans function in kwset.c in grep 2.19 through 2.21 allows local users to cause a denial of service (out-of-bounds heap read and crash) via crafted input when using the -F option."}}]},[...],{"product":"sqlite","version":"3.7.17","is_vulnerable":false,"is_false_positive":false,"vulnerabilities":[]}]},"prog_lang_dependencies":{"vuln_dependencies":1,"dependencies_details":{"java":[],"js":[],"nodejs":[],"php":[],"python":[{"product":"lxml","version":"1.0.1","product_file_path":"/opt/jboss/python/lxml.1.0.1.py","is_vulnerable":true,"is_false_positive":false,"vulnerabilities":[{"CVE-2014-3146":{"cveid":"CVE-2014-3146","cvss_access_complexity":"Medium","cvss_access_vector":"Network","cvss_authentication":"None required","cvss_availability_impact":"None","cvss_base":4.3,"cvss_confidentiality_impact":"None","cvss_exploit":8.6,"cvss_impact":2.9,"cvss_integrity_impact":"Partial","cvss_vector":["AV:N","AC:M","Au:N","C:N","I:P","A:N"],"cweid":"CWE-0","mod_date":"14-04-2015","pub_date":"14-05-2014","summary":"Incomplete blacklist vulnerability in the lxml.html.clean module in lxml before 3.3.5 allows remote attackers to conduct cross-site scripting (XSS) attacks via control characters in the link scheme to the clean_html function."}}]}],"ruby":[]}}}}

Dagda API server response

{"err": 500, "msg": "Internal Server Error"}

Post without format

curl http://localhost:5000/v1/history/image:test -X POST -d {'test':1, 'hola':2}

Dagda API server response
{"err": 500, "msg": "Internal Server Error"}

Dagda Server Error

In both cases, checking the logs of the server I got the same error.

<2019-02-27 13:27:42,509> <ERROR> <DagdaServer> <app> <log_exception:1761> <Exception on /v1/history/image:test [POST]>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.5/dist-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/a2/dagda/dagda/api/service/history.py", line 52, in post_image_analysis_to_the_history
    data = json.loads(request.data.decode('utf-8'))
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Performance issue - Database population process

Short description

Sometimes, the first time the database population process runs, this process takes more than 20 minutes until its completion:

Solution

A possible solution for this issue would be run in a parallel way the gathering of the different vulnerabilities info sources.

Bug in _untar_layers() function

Short description

I think there is a bug in dagda/analysis/static/util/util.py :

def _untar_layers(dir, layers):
output = {}

# Untar layer filesystem bundle
for layer in layers:
    tarfile = TarFile(dir + "/" + layer)
    for member in tarfile.getmembers():
        output[member.name] = member
for member_name in output:
    try:
        tarfile.extract(output[member_name], path=dir, set_attrs=False)
    except (ValueError, ReadError) as ex:
        if InternalServer.is_debug_logging_enabled():
            message = "Unexpected exception of type {0} occurred while untaring the docker image: {1!r}" \
                .format(type(ex).__name__, ex.get_message() if type(ex).__name__ == 'DagdaError' else ex.args)
            DagdaLogger.get_logger().debug(message)
    except PermissionError as ex:
        message = "Unexpected error occurred while untaring the docker image: " + \
                  "Operation not permitted on {0!r}".format(member_name)
        DagdaLogger.get_logger().warn(message)

# Clean up
for layer in layers:
    clean_up(dir + "/" + layer[:-10])

Solution - It should be like this:

def _untar_layers(dir, layers):

# Untar layer filesystem bundle
for layer in layers:
    tarfile = TarFile(dir + "/" + layer)
    output = {}
    for member in tarfile.getmembers():
        output[member.name] = member
    for member_name in output:
        try:
            tarfile.extract(output[member_name], path=dir, set_attrs=False)
        except (ValueError, ReadError) as ex:
            if InternalServer.is_debug_logging_enabled():
                message = "Unexpected exception of type {0} occurred while untaring the docker image: {1!r}" \
                    .format(type(ex).__name__, ex.get_message() if type(ex).__name__ == 'DagdaError' else ex.args)
                DagdaLogger.get_logger().debug(message)
        except PermissionError as ex:
            message = "Unexpected error occurred while untaring the docker image: " + \
                    "Operation not permitted on {0!r}".format(member_name)
            DagdaLogger.get_logger().warn(message)

# Clean up
for layer in layers:
    clean_up(dir + "/" + layer[:-10])

Need to scan a local image.

Short description

I was trying to scan a local image by the name nmap-dist:latest on my machine.

Reproduction steps

  1. Start Dagda server with the parameters '...'
  2. I downloaded the vulnerabilities by 'python3 dagda.py vuln --init'
    and it showed the status updated after a while.
  3. 'python3 dagda.py check --docker_image nmap-dist:latest'
  4. 'python3 dagda.py history nmap-dist:latest --id 5ce791385ea27b39940ecfeb'

Expected results

I expected that by analyzing the docker image 'nmap-dist' and showing the report by using the command 'python3 dagda.py history nmap-dist:latest --id 5ce791385ea27b39940ecfeb' it would give a detailed analysis.

Actual results

Instead of 'detailed ananlysis', what I saw was that

{
    "err": 404,
    "msg": "History not found"
}

happened instead.

On which platforms did you notice this:

Please complete the following information:

  • ubuntu:18.04
  • OS Version: Linux esypc5 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:46:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Python version: [Python 3.6.7]
  • Docker version 18.09.6, build 481bc77
  • MongoDB version
    db version v3.6.3
    git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
    OpenSSL version: OpenSSL 1.1.0g 2 Nov 2017
    allocator: tcmalloc
    modules: none
    build environment:
    distarch: x86_64
    target_arch: x86_64

Backtrace


Solution

Dagda starting with custom mongoDB crashes

Short description

First I must say that this is an impressive piece of software ! Good job !


I tried to connect Dagda to my own mongoDB database but it crashes.

Reproduction steps

That's what I did :

docker pull mongo
docker run -d -p 27017:27017 mongo

Next I populated the database myself (not using vuln init) though the error appears with the db populated or not.

Finally I started Dagda using the 3grander/dagda image :
docker run 40c9973099f7 start -m 192.168.99.100 -mp 27017

Expected results

I expected that Dagda server would be up and running and that I could query it using either CLI or the HTTP API

Actual results

Crashed immediately after starting

On which platforms did you notice this:

  • OS: Windows 10
  • Using docker for windows and the 3grander/dagda image
  • Docker version 18.06.1-ce, build e68fc7a

Backtrace

<2018-11-09 08:24:45,218> <ERROR> <DagdaServer> <docker_driver> <__init__:39> <Error while fetching Docker server API version: Assuming Travis CI tests.>
<2018-11-09 08:24:45,556> <WARNING> <DagdaServer> <sysdig_falco_monitor> <pre_check:83> <I'm running inside a docker container, so I can't check if the kernel headers are installed in the host operating system. Please, review it!!>
<2018-11-09 08:24:45,557> <ERROR> <DagdaServer> <dagda_server> <run:103> <Error while fetching Docker server API version.>
<2018-11-09 08:24:45,558> <WARNING> <DagdaServer> <dagda_server> <run:104> <Runtime behaviour monitor disabled.>
 * Serving Flask app "api.dagda_server" (lazy loading)
 * Environment: production
<2018-11-09 08:24:45,561> <ERROR> <DagdaServer> <dagda> <main:46> <'NoneType' object has no attribute 'events'>
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
Traceback (most recent call last):
  File "dagda.py", line 35, in main
    r = execute_dagda_cmd(cmd=cmd, args=parsed_args)
  File "/opt/app/cli/dagda_cli.py", line 43, in execute_dagda_cmd
    ds.run()
  File "/opt/app/api/dagda_server.py", line 92, in run
    docker_daemon_events_monitor.run()
  File "/opt/app/analysis/runtime/docker_events_monitor.py", line 45, in run
    for event in self.docker_driver.docker_events():
  File "/opt/app/driver/docker_driver.py", line 114, in docker_events
    return self.cli.events()
AttributeError: 'NoneType' object has no attribute 'events'

Solution

I don't know what I am doing wrong !

docker errors :Multiple IDs found with provided prefix

Short description

I was trying to start Dagda server.

Reproduction steps

$ python3 dagda.py start -p 5003 -d

Actual results

$ python3 dagda.py start -p 5003 -d
Serving on http://localhost:5003
<2019-03-06 21:21:57,188> <ERROR> <DagdaServer> <dagda> <main:46> <500 Server Error: Internal Server Error ("Multiple IDs found with provided prefix: 09b61f6b45dde38ebd925697f418386f8f01ccf3029b08985b65a11eebbcb91e")>
Traceback (most recent call last):
  File "/home/sxue/anaconda3/lib/python3.7/site-packages/docker/api/client.py", line 246, in _raise_for_status
    response.raise_for_status()
  File "/home/sxue/anaconda3/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.37/containers/0/stop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dagda.py", line 35, in main
    r = execute_dagda_cmd(cmd=cmd, args=parsed_args)
  File "/home/sxue/project/dagda/dagda/cli/dagda_cli.py", line 43, in execute_dagda_cmd
    ds.run()
  File "/home/sxue/project/dagda/dagda/api/dagda_server.py", line 101, in run
    self.sysdig_falco_monitor.pre_check()
  File "/home/sxue/project/dagda/dagda/analysis/runtime/sysdig_falco_monitor.py", line 97, in pre_check
    self.docker_driver.docker_stop(container_id)
  File "/home/sxue/project/dagda/dagda/driver/docker_driver.py", line 110, in docker_stop
    self.cli.stop(container=container_id)
  File "/home/sxue/anaconda3/lib/python3.7/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/home/sxue/anaconda3/lib/python3.7/site-packages/docker/api/container.py", line 1134, in stop
    self._raise_for_status(res)
  File "/home/sxue/anaconda3/lib/python3.7/site-packages/docker/api/client.py", line 248, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/home/sxue/anaconda3/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("Multiple IDs found with provided prefix: 09b61f6b45dde38ebd925697f418386f8f01ccf3029b08985b65a11eebbcb91e")

On which platforms did you notice this:

Please complete the following information:

  • OS: Ubuntu16.04
  • OS Version: [e.g. (~$ uname -a)]
  • Python version: Python 3.7.0
  • Docker version: Version: 18.03.0-ce
  • MongoDB version : 4.0.6(lastest)

Btw, is there any log file for dagda, I run the command with option -d, the bug log print on the terminal.

Thanks for your help :)

unable to start dagda

Short description

I was trying to do start dagda with 'sudo python3 dagda start', but nothing happened in ubuntu. almost like #29:failed to start dagda, but #29 not finished and closed

Reproduction steps

  1. install docker CE
  2. install pip3
  3. install sysdig (manual)
  4. install requirements.txt
  5. install mongodb
  6. install os kernel
  7. sudo python3 dagda.py start
    then dagda printed, 'serving on localhost:5000' and nothing happened(like #29), over 1 hour. In this situation, i typed 'netstat -an | grep 5000' on the other terminal, and it showed 127.0.0.1 LISTEN.
    I also tried after 'serving on local host:5000' without control+c , export DAGDA_HOST='127.0.0.1' export DAGDA_PORT=5000 python3 dagda.py vuln --init, then nothing happened and i typed 'sudo python3 dagda.py vuln --init_status' on the other terminal, then <dagda_cli><_get_dagda_base_url:174><DAGDA_HOST environment variable is not set.> . . error came out. (also tried when nothing happened after start, typed all commands that export host,port, vuln init, and status on the other terminal, but same error came out)

and these are output of sudo ps -ef
.
.
root 8313 8188 0 09:01 pts/0 00:00:00 sudo python3 dagda.py start
root 8314 8313 0 09:01 pts/0 00:00:01 python3 dagda.py start
root 8319 8314 0 09:01 pts/0 00:00:00 /usr/bin/python3 -c from mult
joon 8320 7696 0 09:01 tty2 00:00:00 /usr/lib/deja-dup/deja-dup-mo
root 8324 8314 0 09:01 pts/0 00:00:00 python3 dagda.py start
root 8325 8314 0 09:01 pts/0 00:00:00 python3 dagda.py start
root 8326 8314 0 09:01 pts/0 00:00:00 python3 dagda.py start
root 10255 2 0 09:02 ? 00:00:00 [kworker/0:1-eve]
root 10287 680 0 09:02 ? 00:00:00 containerd-shim -namespace mo
root 10316 10287 0 09:02 ? 00:00:03 falco -pc -o json_output=true
.
.
i can't find what's wrong in my environment settings , help me

On which platforms did you notice this:

Please complete the following information:

  • OS: Ubuntu [18.04.2 amd 64]
  • OS Version: [(~$ uname -r)]
  • Python version: [Python 3.4]
  • Docker version [18.09.6-ce]
  • MongoDB version [4.0.9]

Vulnerabilities database is not being populated

Short description

I was trying to test dagda.

'dagda.py vuln --product openldap --product_version 2.2.20'

Reproduction steps

  1. Started mongo with docker run -d -p 27017:27017 mongo.
    0.1. Checked mongo is running with:
$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                      NAMES
4c71cbb9f6b4        mongo               "docker-entrypoint.s…"   About an hour ago   Up About an hour    0.0.0.0:27017->27017/tcp   dreamy_snyder
  1. Start Dagda server as 'python3 dagda.py start'
    1.1. The command never returns. The server seems to be running.
  2. From another shell session, export environmental variables then type the Dagda CLI command 'python3 dagda.py vuln --init'
    2.1. Init returns immediately.
    2.2. Connecting to the mongo shows:
> show databases;
admin          0.000GB
config         0.000GB
local          0.000GB
vuln_database  0.000GB

then

> use vuln_database
switched to db vuln_database
> db.init_db_process_status.find()
{ "_id" : ObjectId("5b190d4e1d41c82b38d1462b"), "timestamp" : 1528368462.684211, "status" : "Initializing" }
>
  1. Execute: 'python3 dagda.py vuln --product openldap --product_version 2.2.20'
  2. See response:
{
    "err": 500,
    "msg": "Internal Server Error"
}

Expected results

I expected to see the report as shown in README would be printed out.

Actual results

{
    "err": 500,
    "msg": "Internal Server Error"
}

On which platforms did you notice this:

Please complete the following information:

  • OS: Ubuntu
  • OS Version: Linux ubuntu 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • Python version: Python 3.5.2
  • Docker version: 18.03.1-ce, build 9ee9f40
  • MongoDB version: 3.6.5

Backtrace

[2018-06-08 09:16:30,739] ERROR in app: Exception on /v1/vuln/products/openldap/2.2.20 [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.5/dist-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/shmukler/dagda/dagda/api/service/vuln.py", line 57, in get_vulns_by_product_and_version
    vulns = InternalServer.get_mongodb_driver().get_vulnerabilities(product, version)
  File "/home/shmukler/dagda/dagda/driver/mongodb_driver.py", line 325, in get_vulnerabilities
    for bid in bid_cursor:
  File "/usr/local/lib/python3.5/dist-packages/pymongo/cursor.py", line 1090, in next
    if len(self.__data) or self._refresh():
  File "/usr/local/lib/python3.5/dist-packages/pymongo/cursor.py", line 1012, in _refresh
    self.__read_concern))
  File "/usr/local/lib/python3.5/dist-packages/pymongo/cursor.py", line 905, in __send_message
    helpers._check_command_response(doc['data'][0])
  File "/usr/local/lib/python3.5/dist-packages/pymongo/helpers.py", line 205, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: text index required for $text query (no such collection 'vuln_database.bid')
<2018-06-08 09:16:30,739> <ERROR> <DagdaServer> <app> <Exception on /v1/vuln/products/openldap/2.2.20 [GET]>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.5/dist-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/shmukler/dagda/dagda/api/service/vuln.py", line 57, in get_vulns_by_product_and_version
    vulns = InternalServer.get_mongodb_driver().get_vulnerabilities(product, version)
  File "/home/shmukler/dagda/dagda/driver/mongodb_driver.py", line 325, in get_vulnerabilities
    for bid in bid_cursor:
  File "/usr/local/lib/python3.5/dist-packages/pymongo/cursor.py", line 1090, in next
    if len(self.__data) or self._refresh():
  File "/usr/local/lib/python3.5/dist-packages/pymongo/cursor.py", line 1012, in _refresh
    self.__read_concern))
  File "/usr/local/lib/python3.5/dist-packages/pymongo/cursor.py", line 905, in __send_message
    helpers._check_command_response(doc['data'][0])
  File "/usr/local/lib/python3.5/dist-packages/pymongo/helpers.py", line 205, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: text index required for $text query (no such collection 'vuln_database.bid')

Solution

Not known

vuln doesn't show CVE which is present in securityfocus and NVD

Short description

I am trying to request information with dagda about CVE-2018-15727 which is present in securityfocus and NVD databases and I am getting "CVE not found"

{
    "err": 404,
    "msg": "CVE not found"
}

Reproduction steps

  1. Start Dagda server with docker-compose up
  2. Populate DB with python dagda.py vuln --init and wait until it is finished.
    {
      "status": "Updated",
      "timestamp": "2019-02-15 07:48:54.802088"
    }
    
  3. Point dagda client to server
  4. Type the Dagda CLI command python dagda.py vuln --cve CVE-2018-15727

Expected results

I expected information about CVE. I can get information about other CVE's just fine. For example:

$ python dagda.py vuln --cve CVE-2018-12099
[
    {
        "product": "grafana",
        "vendor": "grafana",
        "version": "5.2.0",
        "year": 2018
    }
]

Actual results

{
    "err": 404,
    "msg": "CVE not found"
}

On which platforms did you notice this:

  • OS: Gentoo
  • OS Version: 4.20.5-gentoo
  • Python version: 3.6.6
  • Docker version: 18.09.2
  • MongoDB version: v4.0.6

Backtrace

During startup I have noticed:

dagda     | <2019-02-15 08:16:27,116> <ERROR> <DagdaServer> <dagda> <main:46> <500 Server Error: Internal Server Error ("Multiple IDs found with provided prefix: 9")>
dagda     | Traceback (most recent call last):
dagda     |   File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 246, in _raise_for_status
dagda     |     response.raise_for_status()
dagda     |   File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
dagda     |     raise HTTPError(http_error_msg, response=self)
dagda     | requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.39/containers/9/stop
dagda     | 
dagda     | During handling of the above exception, another exception occurred:
dagda     | 
dagda     | Traceback (most recent call last):
dagda     |   File "dagda.py", line 35, in main
dagda     |     r = execute_dagda_cmd(cmd=cmd, args=parsed_args)
dagda     |   File "/opt/app/cli/dagda_cli.py", line 43, in execute_dagda_cmd
dagda     |     ds.run()
dagda     |   File "/opt/app/api/dagda_server.py", line 101, in run
dagda     |     self.sysdig_falco_monitor.pre_check()
dagda     |   File "/opt/app/analysis/runtime/sysdig_falco_monitor.py", line 97, in pre_check
dagda     |     self.docker_driver.docker_stop(container_id)
dagda     |   File "/opt/app/driver/docker_driver.py", line 110, in docker_stop
dagda     |     self.cli.stop(container=container_id)
dagda     |   File "/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
dagda     |     return f(self, resource_id, *args, **kwargs)
dagda     |   File "/usr/local/lib/python3.6/site-packages/docker/api/container.py", line 1134, in stop
dagda     |     self._raise_for_status(res)
dagda     |   File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 248, in _raise_for_status
dagda     |     raise create_api_error_from_http_exception(e)
dagda     |   File "/usr/local/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
dagda     |     raise cls(e, response=response, explanation=explanation)
dagda     | docker.errors.APIError: 500 Server Error: Internal Server Error ("Multiple IDs found with provided prefix: 9")

But application works fine(-ish), there is no errors when querying for CVE's

Dadga open multiple threads

Hi @eliasgranderubio,

I've a very strange issue. When i run python3 dagda.py start and then, the vulnerability database init command, dadga starts to open multiple threads:

root     11807  0.1  0.7 474504 14652 pts/0    D+   12:32   0:01 python3 dagda.py start
root     11813 20.1  1.6 944704 31060 pts/0    Rl+  12:33   3:02 python3 dagda.py start
root     11814  0.0  0.4 399248  8472 pts/0    Sl+  12:33   0:00 python3 dagda.py start
root     13242  0.0  0.0  17956  1612 ?        S    12:53   0:00 /bin/bash /usr/bin/falco-probe-loader
root     13256  0.1  0.2  20260  4072 ?        S    12:53   0:00 /bin/bash /usr/sbin/dkms autoinstall --kernelver 3.10.0-514.16.1.el7.x86_64
root     13340  0.0  0.1  20308  3644 ?        S    12:53   0:00 /bin/bash /usr/sbin/dkms autoinstall --kernelver 3.10.0-514.16.1.el7.x86_64
root     13567  0.0  0.1  20308  3188 ?        S    12:53   0:00 /bin/bash /usr/sbin/dkms autoinstall --kernelver 3.10.0-514.16.1.el7.x86_64
root     13568  0.0  0.1  20308  3272 ?        S    12:53   0:00 /bin/bash /usr/sbin/dkms autoinstall --kernelver 3.10.0-514.16.1.el7.x86_64

saturating the OS. I cancel the dadga start execution:

$ python3 dagda.py start

^C
Process ForkPoolWorker-94:
Process ForkPoolWorker-92:
Process ForkPoolWorker-59:
Process ForkPoolWorker-41:
Process ForkPoolWorker-19:
Process ForkPoolWorker-100:
Process ForkPoolWorker-99:
Process ForkPoolWorker-98:
Process ForkPoolWorker-97:
Process ForkPoolWorker-96:
Process ForkPoolWorker-95:
Process ForkPoolWorker-93:
Process ForkPoolWorker-91:
Process ForkPoolWorker-90:
Process ForkPoolWorker-89:
Process ForkPoolWorker-88:
Process ForkPoolWorker-87:
Process ForkPoolWorker-86:
Process ForkPoolWorker-85:
Process ForkPoolWorker-84:
Process ForkPoolWorker-83:
Process ForkPoolWorker-82:
Process ForkPoolWorker-81:
Process ForkPoolWorker-80:
Process ForkPoolWorker-79:
Process ForkPoolWorker-78:
Process ForkPoolWorker-77:
Process ForkPoolWorker-76:
Process ForkPoolWorker-75:
Process ForkPoolWorker-74:
Process ForkPoolWorker-73:
Process ForkPoolWorker-72:
Process ForkPoolWorker-71:
Process ForkPoolWorker-70:
Process ForkPoolWorker-69:
Process ForkPoolWorker-68:
Process ForkPoolWorker-66:
Process ForkPoolWorker-65:
Process ForkPoolWorker-64:
Process ForkPoolWorker-63:
Process ForkPoolWorker-62:
Process ForkPoolWorker-61:
Process ForkPoolWorker-60:
Process ForkPoolWorker-58:
Process ForkPoolWorker-57:
Process ForkPoolWorker-56:
Process ForkPoolWorker-55:
Process ForkPoolWorker-54:
Process ForkPoolWorker-53:
Process ForkPoolWorker-52:
Process ForkPoolWorker-51:
Process ForkPoolWorker-50:
Process ForkPoolWorker-49:
Process ForkPoolWorker-47:
Process ForkPoolWorker-46:
Process ForkPoolWorker-45:
Process ForkPoolWorker-44:
Process ForkPoolWorker-43:
Process ForkPoolWorker-42:
Process ForkPoolWorker-40:
Process ForkPoolWorker-39:
Process ForkPoolWorker-38:
Process ForkPoolWorker-37:
Process ForkPoolWorker-36:
Process ForkPoolWorker-35:
Process ForkPoolWorker-34:
Process ForkPoolWorker-10:
Process ForkPoolWorker-48:
Process ForkPoolWorker-67:
Process ForkPoolWorker-33:
Process ForkPoolWorker-3:
Process ForkPoolWorker-32:
Process ForkPoolWorker-4:
Process ForkPoolWorker-16:
Process ForkPoolWorker-7:
Process ForkPoolWorker-15:
Process ForkPoolWorker-8:
Process ForkPoolWorker-21:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Process ForkPoolWorker-24:
Traceback (most recent call last):
Process ForkPoolWorker-23:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Process ForkPoolWorker-5:
Process ForkPoolWorker-6:
Process ForkPoolWorker-9:
Traceback (most recent call last):
Process ForkPoolWorker-17:
Traceback (most recent call last):
Traceback (most recent call last):

Any idea that what happen? Thanks in advance!

Dagda dont show CVE detail

Hi @eliasgranderubio,

Following the API documentation, i try to view the report information:

# curl http://localhost:5000/v1/history/registry/test | python -m json.tool

[
    {
        "id": "58db8ee208540e14f3c9f58b",
        "image_name": "registry/test",
        "static_analysis": {
            "os_packages": {
                "ok_os_packages": 140,
                "os_packages_details": [
                    {
                        "is_vulnerable": true,
                        "product": "glibc-common",
                        "version": "2.17",
                        "vulnerabilities": [
                            "BID-68006",
                            "BID-70596",
                            "BID-72602",
                            "BID-72603",
                            "BID-76793",
                            "BID-77098",
                            "BID-81469",
                            "BID-82244",
                            "BID-83265",
                            "BID-83275",
                            "BID-83277",
                            "BID-85732",
                            "BID-92257"
                        ]
                    },
                    {
                        "is_vulnerable": true,
                        "product": "glibc",
                        "version": "2.17",
                        "vulnerabilities": [
                            "CVE-2012-4412",
                            "CVE-2012-4424",
                            "CVE-2013-0242",
                            "CVE-2013-1914",
                            "CVE-2013-2207",
                            "CVE-2013-4237",
                            "CVE-2013-4332",
                            "CVE-2013-4458",
                            "CVE-2013-4788",
                            "CVE-2014-0475",
                            "CVE-2014-6040",
                            "CVE-2015-0235",
                            "CVE-2015-7547",
                            "BID-68006",
                            "BID-70596",

but i can't view the CVE details as you shown in the REST API documentation:

{
      "id": "58667994ed253915723c50e7",
      "image_name": "jboss/wildfly",
      "status": "Completed",
      "timestamp": "2016-12-14 13:17:12.802486",
      "static_analysis": {
         "os_packages": {
            "total_os_packages": 182,
            "vuln_os_packages": 41,
            "ok_os_packages": 141,
            "os_packages_details": [
               {
                  "product": "grep",
                  "version": "2.20",
                  "is_vulnerable": true,
                  "vulnerabilities": [
                     {
                        "CVE-2015-1345": {
                           "cveid": "CVE-2015-1345",
                           "cvss_access_complexity": "Low",
                           "cvss_access_vector": "Local access",
                           "cvss_authentication": "None required",
                           "cvss_availability_impact": "Partial",
                           "cvss_base": 2.1,
                           "cvss_confidentiality_impact": "None",
                           "cvss_exploit": 3.9,
                           "cvss_impact": 2.9,
                           "cvss_integrity_impact": "None",
                           "cvss_vector": [
                              "AV:L",
                              "AC:L",
                              "Au:N",
                              "C:N",
                              "I:N",
                              "A:P"
                           ],

Maybe am I forgetting something? Thanks in advance.

Performance issue - Docker image evaluation process

Short description

When the docker image evaluation process runs, this process take too much time and it could be optimized:

Solution

After evaluation the source code, a possible solution for this issue would be to run in a parallel way some analysis steps.

Performance:Is it normal to check image jboss/wildfly around 40 minutes?

Short description

I was trying to analyze docker containers jboss/wildfly followed readme doc and it took about 40 minutes, is it normal?

Issue 34 mention that

both dependencies analysis and malware analysis run in a parallel way into the docker image evaluation process.

Reproduction steps

python3 dagda.py check --docker_image jboss/wildfly

Thank you.

choice of port 5000 conflicts with standard docker registry port

Short description

Quickly starting up dagda using the docker-compose.yml file fails on a host that is also running a persistent docker registry container (https://hub.docker.com/_/registry). This is a common approach as described in the Docker documentation: https://docs.docker.com/registry/deploying/. The dagda port should perhaps be changed to something else.

Reproduction steps

Per instructions, run docker-compose build && docker-compose up

Expected results

A dagda system starts up without intervention.

Actual results

ERROR: for dagda Cannot start service dagda: b'driver failed programming external connectivity on endpoint dagda (a79806fca4ca37a8b41b12ff9cdbfc82563691c8370b2e048750c347b78401a5): Bind for 0.0.0.0:5000 failed: port is already allocated'

On which platforms did you notice this:

Please complete the following information:

  • OS: Ubuntu
  • OS Version: Linux myhost 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Python version: from image 3grander/dagda:0.8.0
  • Docker version: Docker version 18.09.2, build 6247962
  • MongoDB version: from image mongo:latest

Backtrace

...

Solution

Changed all port 5000 references in docker-compose to a different port.

Stop sysdig/falco failed when restart dagda server.

Short description

If Dagda server is started twice, it will fail like issue48.

Reproduction steps

  1. start dagda server
    python3 dagda.py start -p 5003 -d
  2. stop it with CTRL + C
  3. start another dagda server
    python3 dagda.py start -p 5000 -d

Expected results

I expected that the dagda server start successfully both.

Actual results

<2019-03-12 15:09:41,921> <DEBUG> <DagdaServer> <sysdig_falco_monitor> <pre_check:95> <737948e77328251a8c0555dcf9d6767ba0eea108c984e860aeab54e2c07bfc44>
<2019-03-12 15:09:41,921> <DEBUG> <DagdaServer> <docker_driver> <docker_stop:110> <7>
<2019-03-12 15:09:41,926> <ERROR> <DagdaServer> <dagda> <main:46> <500 Server Error: Internal Server Error ("Multiple IDs found with provided prefix: 737948e77328251a8c0555dcf9d6767ba0eea108c984e860aeab54e2c07bfc44")>
Traceback (most recent call last):  File "/home/sxue/project/dagda/dagdaenv/lib/python3.7/site-packages/docker/api/client.py", line 246, in _raise_for_status
    response.raise_for_status()
  File "/home/sxue/project/dagda/dagdaenv/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.37/containers/7/stop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dagda/dagda.py", line 35, in main    r = execute_dagda_cmd(cmd=cmd, args=parsed_args)
  File "/home/sxue/project/dagda/dagda/cli/dagda_cli.py", line 43, in execute_dagda_cmd
    ds.run()  File "/home/sxue/project/dagda/dagda/api/dagda_server.py", line 101, in run
    self.sysdig_falco_monitor.pre_check()
  File "/home/sxue/project/dagda/dagda/analysis/runtime/sysdig_falco_monitor.py", line 98, in pre_check
    self.docker_driver.docker_stop(container_id)
  File "/home/sxue/project/dagda/dagda/driver/docker_driver.py", line 111, in docker_stop
    self.cli.stop(container=container_id)
  File "/home/sxue/project/dagda/dagdaenv/lib/python3.7/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/home/sxue/project/dagda/dagdaenv/lib/python3.7/site-packages/docker/api/container.py", line 1134, in stop
    self._raise_for_status(res)
  File "/home/sxue/project/dagda/dagdaenv/lib/python3.7/site-packages/docker/api/client.py", line 248, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/home/sxue/project/dagda/dagdaenv/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("Multiple IDs found with provided prefix: 737948e77328251a8c0555dcf9d6767ba0eea108c984e860aeab54e2c07bfc44")

I printed the container_id to log, you can seen that the container_id is only the first number when stop container sysdig/falco.
<2019-03-12 14:39:18,563> <DEBUG> <DagdaServer> <docker_driver> <docker_stop:110> <7> [3/1822]

Dagda fails during analysis

Configuration

  • OS: Ubuntu
  • OS Version: Ubuntu 16.04 LTS (Xenial Xerus)
  • Python version: 3.5.2
  • Docker version: 18.09.1, build 4c52b90

Short description

I am trying to setup dagda for study purpose. I did all the steps in the README until the point of analyzing "jboss/wildfly", on which it fails.

Looking at the trace, I am getting the following error:

Backtrace

# python3 dagda.py start
Serving on http://localhost:5000
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/docker/api/client.py", line 246, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.5/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.39/images/create?fromImage=geekduck%2Fclamav&tag=latest

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/cva/dagda/analysis/analyzer.py", line 229, in _threaded_malware
    temp_dir=temp_dir))
  File "/home/ubuntu/cva/dagda/analysis/static/av/malware_extractor.py", line 23, in get_malware_included_in_docker_image
    raw_output = _clamav_run(docker_driver=docker_driver, temp_dir=temp_dir)
  File "/home/ubuntu/cva/dagda/analysis/static/av/malware_extractor.py", line 33, in _clamav_run
    docker_driver.docker_pull('geekduck/clamav')
  File "/home/ubuntu/cva/dagda/driver/docker_driver.py", line 94, in docker_pull
    return self.cli.pull(image_name, tag=tag)
  File "/usr/local/lib/python3.5/dist-packages/docker/api/image.py", line 400, in pull
    self._raise_for_status(response)
  File "/usr/local/lib/python3.5/dist-packages/docker/api/client.py", line 248, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.5/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for geekduck/clamav, repository does not exist or may require 'docker login'")

Concern

It seems like "geekduck/clamav" does not exists on dockerhub anymore. This dependency is breaking the code. It can be observed if I execute the following:

$ docker pull geekduck/clamav 
Using default tag: latest
Error response from daemon: pull access denied for geekduck/clamav, repository does not exist or may require 'docker login'

Unknown Error Arises

Short description

I was trying to analyze a static image by using "Python3 dagda.py check -i . I did get the accepted for analysis message. When I issue the history command it does return an analyzing message, but the server shows an error, forcing it to hang.

On which platforms did you notice this:

Please complete the following information:

  • OS: Centos
  • OS Version: 7
  • Python version: Python 3.6.6
  • Docker version: 18.09.4 build d14af54266
  • MongoDB version: 4.0.8

Backtrace

Serving on http://localhost:5000
<2019-04-11 05:43:51,022> main:46 <'unknown'>
Traceback (most recent call last):
File "dagda.py", line 35, in main
r = execute_dagda_cmd(cmd=cmd, args=parsed_args)
File "/home/user/Desktop/Fresh/dagda/dagda/cli/dagda_cli.py", line 43, in execute_dagda_cmd
ds.run()
File "/home/user/Desktop/Fresh/dagda/dagda/api/dagda_server.py", line 81, in run
self._check_docker_by_image_name(item)
File "/home/user/Desktop/Fresh/dagda/dagda/api/dagda_server.py", line 166, in _check_docker_by_image_name
evaluated_docker_image = analyzer.evaluate_image(item['image_name'], None)
File "/home/user/Desktop/Fresh/dagda/dagda/analysis/analyzer.py", line 120, in evaluate_image
data['static_analysis'] = self.generate_static_analysis(image_name, os_packages, dependencies, malware_binaries)
File "/home/user/Desktop/Fresh/dagda/dagda/analysis/analyzer.py", line 135, in generate_static_analysis
data['prog_lang_dependencies'] = self.generate_dependencies_report(image_name, dependencies)
File "/home/user/Desktop/Fresh/dagda/dagda/analysis/analyzer.py", line 161, in generate_dependencies_report
dep_details[splitted_dep[0]].append(d)
KeyError: 'unknown'

Dagda swarm mode

Short description

Create the new feature Dagda swarm mode for running dagda in a distribute way.

Solution

A possible solution for this issue would be develop a swarm mode which works as docker swarm, I mean, a Dagda agent could join/leave a swarm and these agents would be controled by Dagda server (master) for running static analysis or monitoring the containers no matter where those containers are. For this version it wouldn't be neccessary multiple Dagda server (master) run in a colaborative way.

JSONDecodeError doesn't exist before Python 3.5

When doing a first-time setup (e.g. with REST API endpoint vuln/init), I get the following error:

AttributeError: 'module' object has no attribute 'JSONDecodeError'

It looks like you are expecting json.JSONDecodeError here to fire when the bugtraq data cannot be properly decoded. However, this error class was first added in Python 3.5 (compare 3.4 and 3.5), and the system requirements only state Python3.3+ is required.

Although you could simply bump the version requirement in the readme, would you consider making this compatable with < 3.5 Python? For example, Ubuntu 14.05's builtin Python3 is only 3.4.3.

Dagda has a parsing error with the image packages

Hi @eliasgranderubio,

I'm working in the last issue, i'll give you information throughout the day. In this moment I've a problem with the analysis of a RHEL base image, that generates a totally wrong report because of backporting (https://access.redhat.com/security/updates/backporting):

# docker images
REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE
docker.io/sysdig/falco                     latest              de4c13689a8d        4 days ago          379 MB
docker.io/mongo                            latest              5bc602c0b7fe        10 days ago         360 MB
registry.access.redhat.com/rhel7           latest              41a4953dbf95        5 weeks ago         192.5 MB                                                                    
docker.io/deepfenceio/deepfence_depcheck   latest              d7dc7da5d883        10 weeks ago        915 MB                                                                      
[root@breaknoise dagda]# docker run -it registry.access.redhat.com/rhel7 /bin/bash
[root@584bf64fa883 /]# rpm -qa | grep libc                                                                                                                                         
glibc-common-2.17-157.el7_3.1.x86_64
glibc-2.17-157.el7_3.1.x86_64
libcom_err-1.42.9-9.el7.x86_64
libcap-2.22-8.el7.x86_64
libcap-ng-0.7.5-4.el7.x86_64
libcurl-7.29.0-35.el7.x86_64
[root@584bf64fa883 /]# uname -a                                                                                                                                                    
Linux 584bf64fa883 4.9.13-201.fc25.x86_64 #1 SMP Tue Mar 7 23:47:11 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

as you can see, glibc has 2.17-157 version, but dadga interprets that i've another version (occurs with all packages):

[
    {
        "id": "58dcda9f08540e4ea233aac3",
        "image_name": "registry.access.redhat.com/rhel7:latest",
        "static_analysis": {
            "os_packages": {
                "ok_os_packages": 117,
                "os_packages_details": [
                    {
                        "is_vulnerable": false,
                        "product": "tzdata",
                        "version": "2016j",
                        "vulnerabilities": []
                    },
                    {
                        "is_vulnerable": false,
                        "product": "setup",
                        "version": "2.8.71",
                        "vulnerabilities": []
                    },
                    {
                        "is_vulnerable": false,
                        "product": "basesystem",
                        "version": "10.0",
                        "vulnerabilities": []
                    },
                    {
                        "is_vulnerable": true,
                        "product": "glibc-common",
                        "version": "2.17",
                        "vulnerabilities": [

Finding in total 32 vulnerable packages, but it is not true:

 [root@584bf64fa883 /]# yum info package glibc
 Loaded plugins: ovl, product-id, search-disabled-repos, subscription-manager
 This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
 Installed Packages
 Name        : glibc
 Arch        : x86_64
 Version     : 2.17
 Release     : 157.el7_3.1
 Size        : 13 M
 Repo        : installed
 From repo   : anaconda
 Summary     : The GNU libc libraries
 URL         : http://www.gnu.org/software/glibc/
 License     : LGPLv2+ and LGPLv2+ with exceptions and GPLv2+
 Description : The glibc package contains standard libraries which are used by
             : multiple programs on the system. In order to save disk space and
             : memory, as well as to make upgrading easier, common system code is
             : kept in one place and shared between programs. This particular package
             : contains the most important sets of shared libraries: the standard C
             : library and the standard math library. Without these two libraries, a
             : Linux system will not function.

I paste more info from glibc the package:

[root@584bf64fa883 /]# yumdb info glibc
Loaded plugins: ovl, product-id
glibc-2.17-157.el7_3.1.x86_64
     checksum_data = 91df1204bec843876dbb8c349fcf7cffacea5f8371110787a922190f10b8bd57
     checksum_type = sha256
     from_repo = anaconda
     from_repo_revision = 1487783824
     from_repo_timestamp = 1487783824
     installed_by = 4294967295
     reason = dep
     releasever = 7.3
     ts_install_langs = en
     var_uuid = 2479ed98-9b4a-45b7-a907-97f0182fa875

Dagda does not list running containers in MacOS

I am using Docker-Machine on MacOS. Dagda is unable to check the containers and images that are currently running. Instead, it raises a 404. Is there a way to force Dagda to hit a custom DOCKER_HOST ?

AttributeError: 'NoneType' object has no attribute 'events'

Hi I am not using docker, getting below error. Can you please take a look.

(dagda) ubuntu@ubuntu-Latitude-E6420:~/dagda/dagda$ python3 dagda.py start
<2019-07-06 10:46:02,709> <docker_driver> <init:39>
<2019-07-06 10:46:02,778> main:46 <'NoneType' object has no attribute 'events'>
Traceback (most recent call last):
File "dagda.py", line 35, in main
r = execute_dagda_cmd(cmd=cmd, args=parsed_args)
File "/home/ubuntu/dagda/dagda/cli/dagda_cli.py", line 43, in execute_dagda_cmd
ds.run()
File "/home/ubuntu/dagda/dagda/api/dagda_server.py", line 93, in run
docker_daemon_events_monitor.run()
File "/home/ubuntu/dagda/dagda/analysis/runtime/docker_events_monitor.py", line 45, in run
for event in self.docker_driver.docker_events():
File "/home/ubuntu/dagda/dagda/driver/docker_driver.py", line 114, in docker_events
return self.cli.events()
AttributeError: 'NoneType' object has no attribute 'events'
Serving on http://localhost:5000
<2019-07-06 10:46:02,836> <dagda_server> run:104
<2019-07-06 10:46:02,837> <dagda_server> run:105

Unexpected exception of type BadZipFile occured: ('File is not a zip file',)

Short description

I was trying to initialize vulndb with " vuln --init" cli command.

Reproduction steps

  1. Start Dagda server with the parameters which are default. server: port localhost:5000 and mongo port default.
  2. Type the Dagda CLI command "python3 dagda.py vuln --init"
  3. "python3 dagda.py vuln --init_status" after few minutes

Expected results

I was expecting something like this:

{
"status": "Initialized",
"timestamp": "2018-06-06 05:45:03.871051"
}

Actual results

{
"status": "Unexpected exception of type BadZipFile occured: ('File is not a zip file',)",
"timestamp": "2018-06-06 06:25:29.540764"
}

Backtrace

I don't know where to find it.

Error getting dependencies report when large report

As docker logs output is being used for getting the dependency check report, when this report output is large, it could crash the tool, getting the following ValueError exception when trying to parse the output

Traceback (most recent call last):
 File "dagda.py", line 105, in <module>
   main(DagdaCLIParser())
 File "dagda.py", line 40, in main
   ds.run()
 File "/opt/app/api/dagda_server.py", line 55, in run
   self._check_docker_by_image_name(item)
 File "/opt/app/api/dagda_server.py", line 111, in _check_docker_by_image_name
   evaluated_docker_image = analyzer.evaluate_image(item['image_name'], None)
 File "/opt/app/analysis/analyzer.py", line 27, in evaluate_image
   dependencies = dep_info_extractor.get_dependencies_from_docker_image(self.dockerDriver, image_name)
 File "/opt/app/analysis/static/dependencies/dep_info_extractor.py", line 28, in get_dependencies_from_docker_image
   dependencies_info = raw_info_to_json_array(raw_info)
 File "/opt/app/analysis/static/dependencies/dep_info_extractor.py", line 47, in raw_info_to_json_array
   return json.loads(deleted_debug_info)
 File "/usr/local/lib/python3.4/json/__init__.py", line 318, in loads
   return _default_decoder.decode(s)
 File "/usr/local/lib/python3.4/json/decoder.py", line 343, in decode
   obj, end = self.raw_decode(s, idx=_w(s, 0).end())
 File "/usr/local/lib/python3.4/json/decoder.py", line 359, in raw_decode
   obj, end = self.scan_once(s, idx)
ValueError: Expecting ',' delimiter: line 1 column 36209 (char 36208)

Output problems in the following lines:

  "cve_attack_vector": "NETWORK"
}
{
  "cve_id": "CVE-1999-0142"[INFO] Cleaning up
[INFO] Done

Dadga API error

Hello @eliasgranderubio,

Trying to consume dadga API, i've an error when i call /v1/history:

<2017-03-29 14:14:01,659> <ERROR> <DagdaServer> <app> <Exception on /v1/history [GET]>
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python3.5/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib64/python3.5/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/lib/python3.5/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/usr/lib/python3.5/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3.5/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/scumfrog/Descargas/dagda/dagda/api/service/history.py", line 33, in get_history
    history = InternalServer.get_mongodb_driver().get_docker_image_all_history()
  File "/home/scumfrog/Descargas/dagda/dagda/driver/mongodb_driver.py", line 279, in get_docker_image_all_history
    report['status'] = scan['status']
KeyError: 'status'

not when i try to call to a history with an image ID with

curl localhost:5000/v1/history/image:tag

Any recomendations? Thanks in advance!

Running Dagda in RHEL 7.2 and Fedora 22 not supported

Trying running Dagda on RHEL based distro's (RHEL and Fedora) and following all the requeriments:

# python3.6 -V
Python 3.6.0

Then cloning the project:

# git clone https://github.com/eliasgranderubio/dagda.git
Cloning into 'dagda'...
remote: Counting objects: 944, done.
remote: Total 944 (delta 0), reused 0 (delta 0), pack-reused 944
Receiving objects: 100% (944/944), 631.84 KiB | 519.00 KiB/s, done.
Resolving deltas: 100% (572/572), done.

and installing all python modules required:

# pip3.6 install -r requirements.txt
Requirement already satisfied: pymongo==3.3.1 in /usr/lib64/python3.6/site-packages (from -r requirements.txt (line 1))
Requirement already satisfied: requests==2.11.1 in /usr/lib/python3.6/site-packages (from -r requirements.txt (line 2))
Requirement already satisfied: python-dateutil==2.6.0 in /usr/lib/python3.6/site-packages (from -r requirements.txt (line 3))
Requirement already satisfied: joblib==0.10.3 in /usr/lib/python3.6/site-packages (from -r requirements.txt (line 4))
Requirement already satisfied: docker-py in /usr/lib/python3.6/site-packages (from -r requirements.txt (line 5))
Requirement already satisfied: Flask==0.11.1 in /usr/lib/python3.6/site-packages (from -r requirements.txt (line 6))
Requirement already satisfied: flask-cors==3.0.2 in /usr/lib64/python3.6/site-packages (from -r requirements.txt (line 7))
Requirement already satisfied: PyYAML==3.12 in /usr/lib64/python3.6/site-packages (from -r requirements.txt (line 8))
Requirement already satisfied: six>=1.5 in /usr/lib/python3.6/site-packages (from python-dateutil==2.6.0->-r requirements.txt (line 3))
Requirement already satisfied: docker-pycreds>=0.2.1 in /usr/lib/python3.6/site-packages (from docker-py->-r requirements.txt (line 5))
Requirement already satisfied: websocket-client>=0.32.0 in /usr/lib/python3.6/site-packages (from docker-py->-r requirements.txt (line 5))
Requirement already satisfied: click>=2.0 in /usr/lib/python3.6/site-packages (from Flask==0.11.1->-r requirements.txt (line 6))
Requirement already satisfied: Jinja2>=2.4 in /usr/lib/python3.6/site-packages (from Flask==0.11.1->-r requirements.txt (line 6))
Requirement already satisfied: itsdangerous>=0.21 in /usr/lib/python3.6/site-packages (from Flask==0.11.1->-r requirements.txt (line 6))
Requirement already satisfied: Werkzeug>=0.7 in /usr/lib/python3.6/site-packages (from Flask==0.11.1->-r requirements.txt (line 6))
Requirement already satisfied: MarkupSafe>=0.23 in /usr/lib/python3.6/site-packages (from Jinja2>=2.4->Flask==0.11.1->-r requirements.txt (line 6))

Then running the mongodb container:

# docker run -d -p 27017:27017 mongo
2458d9e7ded96f6200861e8f56d4f84a4dcca2521208f8323289d9e78aaede7e
# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS                      NAMES
2458d9e7ded9        mongo               "/entrypoint.sh mo..."   4 seconds ago       Up 2 seconds               0.0.0.0:27017->27017/tcp   eager_goldwasser

Then, i checked if i've the kernel headers intalled:

# yum -y install kernel-devel-$(uname -r)
Complementos cargados:product-id, search-disabled-repos, subscription-manager
Package kernel-devel-3.10.0-229.11.1.el7.x86_64 already installed and latest version

# /usr/lib/dkms/dkms_autoinstaller start
dkms: running auto installation service for kernel 3.10.0-229.11.1.el7.x86_64
 Done.

and then i running the server with two errors:

# python3.6 dagda.py start
<2017-03-21 11:17:25,200> <ERROR> <DagdaServer> <dagda_server> <Linux distribution not supported yet.>
<2017-03-21 11:17:25,200> <WARNING> <DagdaServer> <dagda_server> <Runtime behaviour monitor disabled.>

If i've the kernel-devel package and i run the dkms_autoinstaller, ¿why i've the runtime behaviour monitor disabled if i follow this https://github.com/eliasgranderubio/dagda/wiki/Troubleshooting#install-the-kernel-headers-in-the-host-os?

I've run dagda in Debian-based distros (like Ubuntu and pure-Debian) without this kind of problems.

Sort Memory errors in Mongodb

Hi again :)

Short description

Trying to extract the reports of statistical analysis, I got this error Sort operation used more than the maximum 33554432 bytes of RAM

I could solve it by opening a shell in the mongdb container and modifying the internalQueryExecMaxBlockingSortBytes parameter to 134 MB with this command:

db.adminCommand({'setParameter':1, 'internalQueryExecMaxBlockingSortBytes':134217728})

A more in-depth description of the space restriction and how to solve it can be found here:
https://mobilemonitoringsolutions.com/memory-errors-mongodb-resolve/

Persistent error

When for other reasons I had to restart the mongo database I lost the configuration and I had this error again because I did not commit to this image.

It would be possible to increase the size of mongo by default, or that somehow through the api of the dagda server configure this parameter?

Backtrace

Dagda server log error

dagda     | 2019-04-03T06:21:45.584498635Z pymongo.errors.OperationFailure: Executor error during find command :: caused by :: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.

Thanks in advance !!

dagda check stuck on Analyzing

Short description

check docker image stuck on "status": "Analyzing"

Reproduction steps

python3 dagda.py start
python3 dagda.py check --docker_image [local_docker_image]

Expected results

I expected that by analyzing the docker image would show the report but after 4 hours no show of any status.

Actual results

Current status :

{
"anomalies": 0,
"image_name": "sonarqube",
"libs_vulns": 0,
"malware_bins": 0,
"os_vulns": 0,
"reportid": "5ba3520bd6ca3f0514fa2833",
"start_date": "2018-09-20 07:53:47.924439",
"status": "Analyzing"
}

On which platforms did you notice this:

running kali in docker with mongodb in docker communicating over exposed port.

Backtrace

<2018-09-20 09:11:47,364> <ERROR> <DagdaServer> <docker_events_monitor> <run:60> <Unexpected exception of type InvalidDocument occured: ("key 'io.github.offensive-security.license' must not contain '.'",)>

Improve the accurate of the analysis

Short description

Improve the accurate of the analysis for avoiding false positives in a static security analysis.

Actual results

Dagda shows false positives and doesn't look like accurate in some analysis. More info in the next link:

Solution

A possible solution for this issue would be review how de packages and dependencies names and versions are collected, and how this information is compared against vulnerability database.

exec sleep error

hi here,

while trying to analyze for testing the docker image traefik:latest, i had these strange bugs comes up while using dagda:

dagda     | 
docker.errors.NotFound: 404 Client Error: Not Found for url: 
http+docker://localunixsocket/v1.28/containers/10851416f569e8489dcc9006baec6751c68e76eeeafb7a233319ff37d6916f32/start ("oci runtime error: container_linux.go:247: 
starting container process caused "exec: \"sleep\": executable file not found in $PATH"

anyone knows how to fix perhaps ?

thanks alot !

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.