intel / meta-security-isafw

meta-security-isafw is an OE layer that allows enabling the Image Security Analysis Framework (isafw) for your image builds.

The primary purpose of isafw is to provide an extensible framework for analysing different security aspects of images during the build process.

The isafw project itself can be found at https://github.com/01org/isafw

The framework supports a number of callbacks (such as process_package(), process_filesystem(), and etc.) that are invoked by the bitbake during different stages of package and image build. These callbacks are then forwarded for processing to the avaliable ISA FW plugins that have registered for these callbacks. Plugins can do their own processing on each stage of the build process and produce security reports.

Dependencies

The meta-security-isafw layer depends on the Open Embeeded core layer:

git://git.openembedded.org/openembedded-core

Usage

In order to enable the isafw during the image build, please add the following line to your build/conf/local.conf file:

INHERIT += "isafw"

Next you need to update your build/conf/bblayers.conf file with the location of meta-security-isafw layer on your filesystem along with any other layers needed. e.g.:

BBLAYERS ?= " \
  /OE/oe-core/meta \
  /OE/meta-security-isafw \
  "

Also, some isafw plugins require network connection, so in case of a proxy setup please make sure to export http_proxy variable into your environment.

In order to produce image reports, you can execute image build normally. For example:

bitbake core-image-minimal

If you are only interested to produce a report based on packages and without building an image, please use:

bitbake -c analyse_sources_all core-image-minimal

Logs

All isafw plugins by default create their logs under the ${LOG_DIR}/isafw-report/ directory, where ${LOG_DIR} is a bitbake default location for log files. If you wish to change this location, please define ISAFW_REPORTDIR variable in your local.conf file.

Patches

Please submit any patches via Github pull requests.

Maintainer: Elena Reshetova [email protected]

meta-security-isafw's Issues

Add a full support for Junit report outcome for automatic diffs in CI

Exception: IndexError: list index out of range

Hello,
used both jethro and master baranches, I'm getting all the time this error :
`The stack trace of python calls that resulted in this exception/failure was:
File: 'analyse_image', lineno: 65, function:
0061: bb.debug(1, 'do image analysis on %s' % rootfsdir)
0062: imageSecurityAnalyser.process_filesystem(fs)
0063:
0064:
*** 0065:analyse_image(d)
0066:
File: 'analyse_image', lineno: 39, function: analyse_image
0035: isafw_config.plugin_blacklist = re.split(r'[,\s]', blacklist)
0036:
0037: imageSecurityAnalyser = isafw.ISA(isafw_config)
0038:
** 0039: pkglist = manifest2pkglist(d)
0040:
0041: kernelconf = d.getVar('STAGING_KERNEL_BUILDDIR', True) + "/.config"
0042:
0043: kernel = isafw.ISA_kernel()
File: 'isafw.bbclass', lineno: 13, function: manifest2pkglist
0009: with open(pkglist, 'w') as foutput:
0010: with open(manifest_file, 'r') as finput:
0011: for line in finput:
0012: items = line.split()
*** 0013: foutput.write(items[0] + " " + items[2] + "\n")
0014:
0015: return pkglist
0016:
0017:
Exception: IndexError: list index out of range

ERROR: Function failed: analyse_image`

Move CFA plugin to be per package basis

Currently CFA plugin runs once after rootfs is done. This can take quite long time for images with a lot of files. If one wants to check and test different compiler flags for a certain package, testing time is not acceptable. So, move CFA plugin to be executed per recipe basis.

Checksum on external file changed

https://github.com/01org/meta-security-isafw/blob/master/recipes-devtools/checksec/checksec_1.5.bb

This recipe downloads the checksec.sh script from a known location. The file format of that has changed recently from UNIX to DOS. This means that the checksum is now different and this recipe will fail.

Package name mismatch between YOCTO project and NVD CVE database.

Found at least for the kernel package:

In NVD CVE Linux Kernel product named as «linux_kernel» with versions in «X.Y.Z» format, while in YOCTO/OpenEmbedded it's name is «linux-yocto» and version «3.19.2+gitAUTOINC+XXXXXXXXXX_XXXXXXXXXX». Therefore check results for such mismatched packets is mean less.

One of the possible approaches to address this issue is to create version mapping between package name/version in YOCTO and package name/version in NVD CVE. Possibly by using package list from analyse_image() phase (or call to manifest2pkglist()).

cve-check-tool is not finding its plugins when running in CI

This is an open issue that waits upon the cve-check-tool maintainer to fix the module loading. Currently modules are loaded from build-time hard-coded path and this breaks in CI using sstate.

CFA: excessive memory usage

In private builds (i.e. can't link to them here) we see excessive memory usage (>50GB) in the bitbake workers which (most likely) executes the CFA process_filesystem() code. Builds never completed without getting killed by the kernel OOM killer and we haven't had time to investigate, so this might be both a problem with the code or the image that is getting analyzed.

However, even if the image is just producing too many reports, the code should be able to handle that without this excessive memory usage.

Offline checks in meta-security layer

Meta-security layer at http://git.yoctoproject.org/cgit/cgit.cgi/meta-security contains several scripts for offline security checks (see package redhat-security). Could you consider adding some of those scripts to the isafw layer or implementing corresponding functionality in existing checks? Thanks!

isafw broken on morty

master is broken on morty since b45750e .

NOTE: Tasks Summary: Attempted 3639 tasks of which 3383 didn't need to be rerun and all succeeded.
ERROR: Execution of event handler 'process_reports_handler' failed
Traceback (most recent call last):
File "/repoclone/output/../meta-security-isafw/classes/isafw.bbclass", line 95, in process_reports_handler(e=<bb.event.BuildCompleted object at 0x7fd95f7fa9d0>):
native_sysroot = dd.expand("${STAGING_DIR}/${BUILD_ARCH}")
> staging_populate_sysroot_dir(target_sysroot, native_sysroot, True, dd)

NameError: global name 'staging_populate_sysroot_dir' is not defined

ERROR: Running idle function
Traceback (most recent call last):
File "/repoclone/poky/bitbake/lib/bb/cooker.py", line 1392, in buildTargetsIdle(server=<ProcessServer(ProcessServer-1, started)>, rq=<bb.runqueue.RunQueue instance at 0x7fd9542d05a8>, abort=False):
if not retval:
> bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runq_fnid), buildname, targets, failures, interrupted), self.data)
self.command.finishAsyncCommand(msg)
File "/repoclone/poky/bitbake/lib/bb/event.py", line 178, in fire(event=<bb.event.BuildCompleted object at 0x7fd95f7fa9d0>, d=<bb.data_smart.DataSmart object at 0x7fd96895f390>):

>    fire_class_handlers(event, d)
     if worker_fire:

File "/repoclone/poky/bitbake/lib/bb/event.py", line 110, in fire_class_handlers(event=<bb.event.BuildCompleted object at 0x7fd95f7fa9d0>, d=<bb.data_smart.DataSmart object at 0x7fd96895f390>):
continue
> execute_handler(name, handler, event, d)

File "/repoclone/poky/bitbake/lib/bb/event.py", line 82, in execute_handler(name='process_reports_handler', handler=<function process_reports_handler at 0x7fd965bf9b90>, event=<bb.event.BuildCompleted object at 0x7fd95f7fa9d0>, d=<bb.data_smart.DataSmart object at 0x7fd96895f390>):
try:
> ret = handler(event)
except (bb.parse.SkipRecipe, bb.BBHandledException):
File "/repoclone/output/../meta-security-isafw/classes/isafw.bbclass", line 95, in process_reports_handler(e=<bb.event.BuildCompleted object at 0x7fd95f7fa9d0>):
native_sysroot = dd.expand("${STAGING_DIR}/${BUILD_ARCH}")
> staging_populate_sysroot_dir(target_sysroot, native_sysroot, True, dd)

NameError: global name 'staging_populate_sysroot_dir' is not defined

Failed again, bail out ...

CFA: invalid index exception

Sometimes, under uncertain circumstances, parsing manifest files fails because a line does not contain two words, leading to this exception (https://ostroproject.org/jenkins/job/build_intel-quark/1997/console):

ERROR: Exception in plugin ISA_LicenseChecker process_report():
03:46:29.670 Traceback (most recent call last):
03:46:29.670   File "/var/lib/jenkins/ostro-worker-19-slot-0-OTusb/ostro-os/meta-security-isafw/lib/isafw/isafw.py", line 133, in call_plugins
03:46:29.670     method(*parameters, **keywords)
03:46:29.670   File "/var/lib/jenkins/ostro-worker-19-slot-0-OTusb/ostro-os/meta-security-isafw/lib/isafw/isaplugins/ISA_la_plugin.py", line 262, in process_report
03:46:29.670     return LicenseChecker.process_report()
03:46:29.670   File "/var/lib/jenkins/ostro-worker-19-slot-0-OTusb/ostro-os/meta-security-isafw/lib/isafw/isaplugins/ISA_la_plugin.py", line 127, in process_report
03:46:29.670     self.process_pkg_list()
03:46:29.670   File "/var/lib/jenkins/ostro-worker-19-slot-0-OTusb/ostro-os/meta-security-isafw/lib/isafw/isaplugins/ISA_la_plugin.py", line 145, in process_pkg_list
03:46:29.670     orig_pkg_name = line.split()[2]
03:46:29.670 IndexError: list index out of range

That was with meta-security-isafw 89d4453.

At the very least we should add better error handling for the input files. Then if those errors trigger, we may get more information about the root cause of the problem.

CVE checker: report fatal errors

Currently the CVE checks fail with:

<testsuite name="CVE_Plugin" tests="3"><testcase classname="ISA_CVEChecker" name="Update of db forced"/><testcase classname="ISA_CVEChecker" name="Failed to fetch https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2002.meta"/><testcase classname="ISA_CVEChecker" name="DB update failure"/></testsuite>

However, that is not detected as error by Jenkins:
https://ostroproject.org/jenkins/view/Code-Analysis/job/code_isafw_reports/31/

Build #462 in that report had the error above but gets shown as "blue" = everything okay.

Make execution of plugins configurable.

Right now, isafw loads and executes all available plugins at the time.
This might be not desired by some developers. e.g. some would be interested in subset of checks.
To improve situation, we need to implement two lists: whitelist and blacklist for plugins.

Algorithm during plugin loading should be something like that:

for plugin in iterate_over_available_plugins():
    if whitelist and plugin not in whitelist:
        continue
    if blacklist and plugin in blacklist:
        continue
    load_plugin(plugin)

Error when trying to execute bitbake -c analyse_sources_all core-image-sato

...
WARNING: Failed to fetch URL http://www.apache.org/dist/subversion/subversion-1.8.13.tar.bz2, attempting MIRRORS if available
ERROR: Error executing a python function in /home/epetmab/git/poky/meta/recipes-devtools/gcc/libgcc-initial_4.9.bb:

The stack trace of python calls that resulted in this exception/failure was:
File: 'do_analyse_sources', lineno: 64, function:
0060:
0061: return
0062:
0063:
*** 0064:do_analyse_sources(d)
0065:
File: 'do_analyse_sources', lineno: 29, function: do_analyse_sources
0025:
0026: fetch = bb.fetch2.Fetch([], d)
0027: for url in fetch.urls:
0028: workdir = d.getVar('ISAFW_WORKDIR', True)
*** 0029: fetch.unpack(workdir, (url,))
0030:
0031: recipe = isafw.ISA_package()
0032: recipe.name = d.getVar('PN', True)
0033: recipe.version = d.getVar('PV', True)
File: '/home/epetmab/git/poky/bitbake/lib/bb/fetch2/init.py', lineno: 1686, function: unpack
1682:
1683: if ud.lockfile:
1684: lf = bb.utils.lockfile(ud.lockfile)
1685:
*** 1686: ud.method.unpack(ud, root, self.d)
1687:
1688: if ud.lockfile:
1689: bb.utils.unlockfile(lf)
1690:
File: '/home/epetmab/git/poky/bitbake/lib/bb/fetch2/init.py', lineno: 1458, function: unpack
1454:
1455: os.chdir(save_cwd)
1456:
1457: if ret != 0:
*** 1458: raise UnpackError("Unpack command %s failed with return value %s" % (cmd, ret), urldata.url)
1459:
1460: if iterate is True:
1461: iterate_urldata = urldata
1462: iterate_urldata.localpath = "%s/%s" % (rootdir, iterate_file)
Exception: UnpackError: Unpack failure for URL: 'http://ftp.gnu.org/gnu/gcc/gcc-4.9.3/gcc-4.9.3.tar.bz2'. Unpack command PATH="/home/epetmab/programming/yocto/isafw_analyse_test/tmp/sysroots/x86_64-linux/usr/bin/x86_64-poky-linux.gcc-cross-initial-x86_64:/home/epetmab/git/poky/scripts:/home/epetmab/programming/yocto/isafw_analyse_test/tmp/sysroots/x86_64-linux/usr/bin/x86_64-poky-linux:/home/epetmab/programming/yocto/isafw_analyse_test/tmp/sysroots/qemux86-64/usr/bin/crossscripts:/home/epetmab/programming/yocto/isafw_analyse_test/tmp/sysroots/x86_64-linux/usr/sbin:/home/epetmab/programming/yocto/isafw_analyse_test/tmp/sysroots/x86_64-linux/usr/bin:/home/epetmab/programming/yocto/isafw_analyse_test/tmp/sysroots/x86_64-linux/sbin:/home/epetmab/programming/yocto/isafw_analyse_test/tmp/sysroots/x86_64-linux/bin:/home/epetmab/git/poky/scripts:/home/epetmab/git/poky/bitbake/bin:/home/epetmab/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/opt/quest/bin:/opt/quest/sbin:/opt/elx/bin:/opt/elx/bin:/home/epetmab/bin:/home/epetmab/bin" bzip2 -dc /home/epetmab/programming/yocto/isafw_analyse_test/downloads/gcc-4.9.3.tar.bz2 | tar x --no-same-owner -f - failed with return value 2

ERROR: Function failed: do_analyse_sources
ERROR: Logfile of failure stored in: /home/epetmab/programming/yocto/isafw_analyse_test/tmp/work/core2-64-poky-linux/libgcc-initial/4.9.3-r0/temp/log.do_analyse_sources.824
ERROR: Task 458 (/home/epetmab/git/poky/meta/recipes-devtools/gcc/libgcc-initial_4.9.bb, do_analyse_sources) failed with exit code '1'
NOTE: Tasks Summary: Attempted 881 tasks of which 354 didn't need to be rerun and 1 failed.
Waiting for 0 running tasks to finish:

Summary: 1 task failed:

END ERROR MSG

Need Warrior release added to layer compatibility

We'll need the Warrior release added to the layer compatibility.
Currently has : LAYERSERIES_COMPAT_security-isafw = "sumo thud"
Need to add Warrior.

I can setup a pull request this weekend - unless someone gets to it before me.

cve-check-tool fails to run

After the recipe-specific-sysroots change was merged into oe-core, the cve-check-tool doesn't appear to run anymore. File build/tmp-glibc/log/isafw-logs/isafw_cvelog says:

cve-check-tool is missing!

If I install cve-check-tool directly on the host, the error goes away. It could be that a different method for installing the tool is needed?