GithubHelp home page GithubHelp logo

mercedes-benz / sechub Goto Github PK

View Code? Open in Web Editor NEW
247.0 11.0 56.0 32.03 MB

SecHub provides a central API to test software with different security tools.

Home Page: https://mercedes-benz.github.io/sechub/

License: MIT License

Shell 3.89% Groovy 0.13% Java 92.95% Go 2.45% Batchfile 0.01% CSS 0.02% HTML 0.45% ABAP 0.01% Ada 0.01% Assembly 0.01% C 0.01% Erlang 0.01% Fortran 0.01% OCaml 0.01% OpenEdge ABL 0.01% Python 0.08% Ruby 0.01% Scheme 0.01% Tcl 0.01% Visual Basic .NET 0.01%
security orchestration api client server secdevops security-tools vulnerability-scanners security-scanner dast

sechub's Introduction

Build status

SecHub

Eugen

The free and open-source security platform SecHub, provides a central API to test software with different security tools. Many free and open-source as well proprietary security tools are supported by SecHub.

SecHub features:

Supported security tools:

Installation

Please visit https://github.com/mercedes-benz/sechub/wiki/ for detailed information.

Documentation

Introduction

SecHub orchestrates various security and vulnerability scanners which can find potential vulnerabilities in sourcecode, binaries or web applications. This enables security, development and operation teams to review and fix security issues. As a result, SecHub improves application security.

SecHub basic architecture overview
                                                   +--------------+
                                              +--> | PDS + Tool A |
                                              |    +--------------+
+--------+                     +---------+    |
| SecHub | ---- scan data ---> | SecHub  | <--+
| Client | <---- report ------ |   API   | <--+
+--------+                     +---------+    |
                                              |    +--------------+
                                              +--> | PDS + Tool B |
                                                   +--------------+

The objective of SecHub is to help secure the software development lifecyle (SDLC) phases: development, deployment and maintenance. From the first written line of code to the application being in production. SecHub can be used to scan the software continuously.

The security tools are categorized into modules which are named after the security testing method they perform: codeScan, licenseScan, secretScan, webScan etc.

Note

The terms SAST (Static Application Security Testing) and DAST (Dynamic Application Security Testing) are intentionally not used for the module names, because the designers feel those terms are vague and difficult to understand for non-security experts. On the other hand, security experts can easily map: codeScan to SAST and webScan to DAST.

Because of the modules, the user only needs to understand what security testing a module performs, rather than to know what specific security tools are used for the actual scan. The user describes in a configuration file what module(s) to use. Or in other words, what security testing methods should be used to test the software with.

To start a scan, the user sends the configuration file to the SecHub server via API. For ease of use SecHub offers a client which calls the REST API for the user. The SecHub client can be used manually or can be integrated into a continuous-integration build pipeline (see below figure).

CI/CD systems or users can scan with SecHub
/------\
| User | ----+
\------/     |            +--------+
             +-- Scan --> |        |
                          | SecHub |
             +-- Scan --> |        |
/-------\    |            +--------+
| CI/CD | ---+
\-------/

On receiving the user request, SecHub creates a job, delegates it to one or more security tools, collects the results and converts all into a report. Next, the user can download the report in a JSON or HTML format.

User Perspective

SecHub is designed to provide an efficient user workflow. The basic idea of SecHub is, that the user has to do as little as possible to execute a security scan. The user has two options:
a) to use the REST API directly
b) to use the SecHub client.
Both can be integrated into a CI/CD pipeline.

Using the REST API requires several steps, which is fine if SecHub needs to be integrated into another software or platform.

However, it is recommended to use the SecHub client.
The SecHub client reduces the workflow to three steps:

SecHub three steps to scan
1. Create a configuration file  // (1)

/------+
| JSON |
+------/

2. Set Credentials // (2)

export SECHUB_USERID=myUserName…
…

3. Scan // (3)

+--------+                 +--------+
|        | ---- scan ----> |        |
| Client |                 | SecHub |
|        | <-- report ---- |        |
+--------+                 +--------+

(1) Create a SecHub configuration file. This step only needs to be done the first time.

(2) Provide the SecHub credentials.
Example:
export SECHUB_USERID=myUserName
export SECHUB_APITOKEN=NTg5YSMkGRkM2Uy00NDJjLTkYTY4NjEXAMPLE
export SECHUB_SERVER=https://sechub.example.com:8443

(3) Scan using sechub scan

Once the scan is finished, the client returns a report.

If the client is used to scan asynchronously it will return a jobUUID which can be used to get the report:

Scan asynchronously
1. Scan asynchronously // (1)

+--------+                     +--------+
|        | --- scanAsync ----> |        |
| Client |                     | SecHub |
|        | <--- jobUUID ------ |        |
+--------+                     +--------+

2. GetReport // (2)

+--------+                     +--------+
|        | --- getReport ----> |        |
| Client |                     | SecHub |
|        | <--- report ------- |        |
+--------+                     +--------+

(1) Scan asynchronously using sechub scanAsync.
(2) Get report sechub -jobUUID <jobUUID> getReport.

In general, the jobUUID can be used to download the report again and again by different users and in different formats.

Report

SecHub collects the scan results from various security tools and converts them into a unified reporting format called: SecHub Report. The advantage is that the user needs to learn only one report format. The json-report below shows how a report can look like:

JSON report example based on a scan of the G101, G103, G304 examples from GoSec.
{
   "result": {
      "count": 4,
      "findings": [
         {
            "id": 1,
            "description": "Potential hardcoded credentials",
            "name": "Potential hardcoded credentials",
            "severity": "HIGH",
            "code": {
               "location": "examples/g101.go",
               "line": 7,
               "column": 9,
               "source": "var password = \"f62e5bcda4fae4f82370da0c6f20697b8f8447ef\""
            },
            "type": "codeScan",
            "cweId": 798
         },
         {
            "id": 2,
            "description": "Use of unsafe calls should be audited",
            "name": "Use of unsafe calls should be audited",
            "severity": "MEDIUM",
            "code": {
               "location": "examples/g103.go",
               "line": 16,
               "column": 21,
               "source": "intPtr = (*int)(unsafe.Pointer(addressHolder))"
            },
            "type": "codeScan",
            "cweId": 242
         },
         {
            "id": 3,
            "description": "Use of unsafe calls should be audited",
            "name": "Use of unsafe calls should be audited",
            "severity": "MEDIUM",
            "code": {
               "location": "examples/g103.go",
               "line": 15,
               "column": 30,
               "source": "addressHolder := uintptr(unsafe.Pointer(intPtr)) + unsafe.Sizeof(intArray[0])"
            },
            "type": "codeScan",
            "cweId": 242
         },
         {
            "id": 4,
            "description": "Use of unsafe calls should be audited",
            "name": "Use of unsafe calls should be audited",
            "severity": "MEDIUM",
            "code": {
               "location": "examples/g103.go",
               "line": 15,
               "column": 56,
               "source": "addressHolder := uintptr(unsafe.Pointer(intPtr)) + unsafe.Sizeof(intArray[0])"
            },
            "type": "codeScan",
            "cweId": 242
         }
      ]
   },
   "messages": [],
   "reportVersion": "1.0",
   "trafficLight": "RED",
   "status": "SUCCESS",
   "jobUUID": "15a96c07-dcf3-4cbc-8d82-0acc9facd3a6"
}

The report can be downloaded in two flavors: JSON and HTML. Both are human readable. The HTML report is self-contained and can be read in any browser. The JSON format is machine readable and can be read by the SecHub plugins.

Mark Findings as False-Positives

There are two major reasons for marking a security finding as false-positive:
- It is an actual false-positive.
- The finding is a false-positive in the context of the application. For example, the application is never deployed to be reachable from the internet.

Regardless the reason, SecHub supports marking findings as false-positives. The marking of false-positives is a SecHub feature and is independent of the security tools used to scan.

SecHub Plugins

The SecHub Plugins improve the user experience by enabling the user to work directly with the SecHub report in the IDE or text editor.

SecHub plugins exist for the following text editors and IDEs:

All plugins are free and open-source software (FOSS) and can be installed directly from within the IDE or text editors.

Modules

Security tools are categorized into modules.

Each module performs a different security testing method:

  • codeScan - scans code or binaries for potential vulnerabilities (weaknesses).
    This includes SAST (static application security testing) and IaC (infrastructure as code).

  • infraScan - scans infrastructure for vulnerabilities.

  • licenseScan - scans code or artifacts for license information.

  • secretScan - scans code or artifacts for secrets (API tokens, certificates, passwords).

  • webScan - scans a deployed web application for vulnerabilities. Also knows as DAST.

codeScan

Alias: Static application security testing (SAST), static code analysis, infrastructure-as-code (IaC) scan

Status: Productive

The codeScan module scans source code or binary artifacts for potential vulnerabilities (weaknesses). To scan the user uploads the code or binary to SecHub. Once the files are uploaded, SecHub delegates the scan to one of many security tools.

PDS-Solutions:
- GoSec
- PMD
- FindSecurityBugs
- Bandit
- Checkmarx SAST (wrapper only)
- … and more

secretScan

Status: Productive

Scans code or artifacts for secrets (API tokens, certificates, passwords).

PDS-Solutions:
- Gitleaks

webScan

Alias: Dynamic application security testing (DAST)

Status: Productive

The webScan module scans running web applications for vulnerabilities. The only requirement is that the web application can be reached by SecHub via network.

PDS-Solutions:
- OWASP ZAP

infraScan

Status: Experimental

The infraScan scans systems in a network.

licenseScan

Status: Experimental

The licenseScan module scans code or artifacts for license information.

PDS-Solutions:
- Scancode
- Tern

Architecture

SecHub is designed to execute hundreds of scans. It can scale horizontally and vertically. It can run on bare-metal, virtual machines, kubernetes or in the cloud.

The smallest useful setup is: a single SecHub server and a single product delegation server (PDS). Those two components are enough to start scanning.

For a larger setup, the number of SecHub server instances can be increased. More PDS instances can be added.
The only requirements to scale SecHub are: a PostgreSQL database and an object store or file share. SecHub and PDS instances use the PostgreSQL database to share information between instances. For example, the job queue is kept in PostgreSQL. In addition,an object store or file share is necessary, so that all SecHub or PDS instances can store/read files.

Regardless of the backend complexity, whether one SecHub server or many are used the workflow for the user stays the same.

For more details about the architecture have a look at the architecture documentation: https://mercedes-benz.github.io/sechub/latest/sechub-architecture.html.

Operations Perspective

One needs to configure the scan tools as well as manage users and projects.
For details please check the operations guide.

REST API

All user and administrative tasks can be done via REST API. SecHub is designed as a RESTful server.

Contributing

We welcome any contributions. If you want to contribute to this project, please read the contributing guide.

Code of Conduct

Please read our Code of Conduct as it is our base for interaction.

License

This project is licensed under the MIT LICENSE.

Provider Information

Please visit https://www.mercedes-benz-techinnovation.com/en/imprint/ for information on the provider.

Notice: Before you use the program in productive use, please take all necessary precautions, e.g. testing and verifying the program with regard to your specific use. The program was tested solely for our own use cases, which might differ from yours.

sechub's People

Contributors

4w31ss3 avatar abdullah-benomar-shahen avatar alexdd avatar ddzikon avatar de-jcup avatar dependabot[bot] avatar dkuerner avatar expp121 avatar fanjups avatar haerter-tss avatar jeeppler avatar jonico avatar jplewa avatar lorriborri avatar masterhanddev avatar mmorel-35 avatar puspel avatar raptor-07 avatar roventu avatar steleow avatar sven-dmlr avatar tragio avatar vivek-prajapatii avatar winzj avatar zigfridus avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sechub's Issues

Provide OpenID connect

At the moment we got only basic authentication - secured by HTTPS.

For better User ID handling, corporate integration etc. we will provide openID connect

Handle empty source uploads in dedicated way

Checkmarx adapter does currently fail when source files are empty.

E.g. :

Caused by: com.daimler.sechub.adapter.AdapterException: __[SECHUB-UID:18b606d1-6729-486f-83d1-50ea0c7634ed]__ CheckmarxAdapterV1:The queuing has failed:Failed to start scanning due to one of following reasons: source folder is empty, all source files are of an unsupported language or file format

On response we got only one finding with text:
"description" : "Security product 'CHECKMARX' failed, so cannot give a correct answer.",

We could handle this in future in one of the following variants:

  1. Show a dedicated finding (warning) that no sources where uploaded
    and do not call any product . So no matter which product this would be always same.
  2. Let it as is - means being product specific response.... not nice
  3. SecHub accepts this always as "green" - no source, no cry...
    So no matter which product this would be always same.

Enable / Disable scheduler job processing + provide Scheduler Status

At the moment a deployment in cluster environment - e.g. kubernetes - has got following problem:

When we a upgrade deployed server versions to next version, the running server instances will be stopped.... so also the JVMs... means also every running SecHub Job inside... After the new deployments these jobs are lost.

The easiest way to handle this, is to integrate a switch option into sechub administration and sechub scheduler.

Usecases:

  • Admin stops scheduling
  • Admin restarts scheduling
  • Admin gets information about scheduling status (stopped,running, amount of waiting scheduled Jobs)

When sheduling is enabled/disabled a notification to admins shall be sent, so it's clear what happend (when disabled no job processing happens...)

Improve CORS handling for anonymous signup

At AnonymousSignupRestController there is a usage of
@CrossOrigin
but without any restriction.

https://spring.io/guides/gs/rest-service-cors/
describes how to setup this.

The best way would be to make the origins configurable. If this is possible by using spring value here should be checked.
Otherwise maybe cors filtering could be an option.

By allowing any location to call this, this could lead to a security problem (maybe)

Checkmarx team id is fix - handle multi tenancy

Currently the teamId used in checkmarx product executor is fix.

Because checkmarx does group users by teamId and has no other option to reduce visibility
of scans, we must change the behaviour.

A project scan should have always its own project team id instead of fix one.

But this could lead either into automated teamId creation for new projects, or if not possible by rest API this will become part of the onboarding process.

Netsparker team id fix - how to handle multi tenancy

Situation

Netsparker has (currently) following permission model:

user1a (=Team1 = Team1 admin)
|_user1b
|_user1c

user2a (=Team2 = Team2 admin)
|_user2b
|_user2c

so when user1a scans something user1b and user1c will have access to all of these results. But user2a,user2b,user2c will not have access.

At the moment we provide only one Team means, when we would give users access to netsparker UI, they will see all results .

Wanted

Every project user should only see its own project results.

Obstacles

One thought solution was to define a team for every sechub project and give access to users
inside this team only.

Unfortunately netsparker users must all have unique email adresses and we got a 1:1 mapping between team admin and users, means when we got a user which is in project1 and project2 we could not manage this situation - except when we add two different email addresses (means accounts) to same person! So a solution by just setting up for netsparker teams for each sechub project will not work!

Solution

Netsparker provides a mechanism called "website groups". Inside these groups you can give explicit permissions. Having a website group defined those scans are no longer visible for all users but only dedicated ones.

Example:

user1a (=Team1 = Team1 admin)
|_user1b
|_user1c
|_website-group1 (contains. test1.example.com)
   |_assign user1b to this group
|_website-group2 (contains. test2.example.com)
   |_assign user2b to this group

In the example before user2b has only access t o scans for test2.example.com.

So this solves our problem and we have no

Consequences

  • If you need to separate results because of need of multi tenancy you are currently forced
    to create website groups (manual) and add users to these website groups (manual)

Continous integration build fails with missing property exception

Continous integration build fails with

groovy.lang.MissingPropertyException: No such property: failed for class: groovy.lang.Binding
	at groovy.lang.Binding.getVariable(Binding.java:63)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:270)
	at org.kohsuke.groovy.sandbox.impl.Checker$6.call(Checker.java:289)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:293)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:269)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:269)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:269)
	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.getProperty(SandboxInvoker.java:29)
	at com.cloudbees.groovy.cps.impl.PropertyAccessBlock.rawGet(PropertyAccessBlock.java:20)
	at WorkflowScript.run(WorkflowScript:62)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:74)
	at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
	at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:66)
	at sun.reflect.GeneratedMethodAccessor170.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:186)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:370)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:93)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:282)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:270)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

Provide Message-Groups and change documentation generator

Currently every MessageID is listed as ONE list, also the PlantUML file overview is a ONE picture.

At the moment this works. But when having some additional messages we will get problems in overview and in readability of document.

So we should introduce a grouping mechanism (maybe another enum, used inside MessageId enum) to make smaller PlantUml views and to have a structured list inside documentation.

Remove Demo Mode Profile and Setup

At the beginning of sechub there was a Profile "DEMO_MODE" for presentation.

This profile must be removed - when somebody wants a demo without having access to real security products, the INTEGRATION_TEST profile can be used.

Improve project time to wait for other projects on a running job

Situation

We got the situation that some security products do not scale good enough and can block other project scans by just one full fat big scan. And when only one instance is running all others must wait...

Very , very painful for other projects.

Wanted

High critical projects (time) should be processes faster, even when there are many existing parts in queue.

How could we achieve this?

Idea 1: Security Environments

An administrator defines something like "security-environment". There exists a default
environment which is used by every project (fallback) not being assigned to a specific environment.

Inside the environment an administrator can define a configuration setup for dedicated products. "e.g. something like 'VIP-project-XYZ' would be defined as an environment and has got not default scan server as target, but a dedicated one.

Only an administrator can change this setup. The setup should be product specific. So every adapter must be able to operate with this settings. Easiest way could be to use a JSON text file which represents products id and a general key value strucuture. On later admin web ui the content should be graphical by providing a key value UI...

Risks/Problems
  • How to provide secrets for products here?

Delete project failed on developer admin ui

Delete project does not work inside admin ui for test scenario3 on integration-test-server

Steps to reproduce:

  • start integration test server
  • start admin ui
  • create test data (scenario3)
  • delete project scenario3_project1

I got following message inside UI:


[EXECUTE] DeleteProjectAction-1566973605364
ERROR:
status code::405, text:,body:
{"status":405,"error":"Method Not Allowed","message":"Request method 'POST' not supported","details":[],"timeStamp":"Wed Aug 28 08:26:50 CEST 2019"}
sent
[DONE]

Change go client, so apitoken can be set by environment entry instead argument

Currently we got same situation for go client as for admin ui, see #8 :
Everybody able to see processes on server side does also see api token used by calling
sechub client.

SECHUB_APITOKEN environment shall be an addition to existing code. It shall be downward compatible. But a warning should be printed out when user is using argument instead of env entry.

Netsparker: change website name generator because HTTP vs. HTTPS problem

Currently website name for netsparker is generated by md5 of complete root target uri.

For example:
http://testserver.your.corp.example.com:8082

but if you have change your website from HTTP only o HTTPS, you will have
https://testserver.your.corp.example.com:8082 which leads to another MD5 value, which leads on next scan to a new created website (in netsparker).

Unfortunately netsparker does not allow this because the former scanned http pages are (correctly) recognized as duplicates.

So we must change the name creation and no longer use
http://testserver.your.corp.example.com:8082
but
testserver.your.corp.example.com:8082
as identifier!

we also will not long use MD5 for names, but use only hostname + _ + port

so name in former example will be:
testserver.your.corp.example.com_8082

when no port is given, we set "DEFAULT"
testserver.your.corp.example.com_default

Automatically remove old scans

About

To prevent full hard drives there should be a option to automatically remove old scan data.

Wanted

As an administrator I can configure a "auto cleanup configuration" and after which age a Job and it's data will be automatically dropped.

Solution

Configuration

Usecases

UC:Administrator updates auto cleanup configuration

Domain:administration

  • REST API for admin domain
  • Stores new configuration inside database
  • An event with message id AUTO_CLEANUP_CONFIGURATION_CHANGEDis triggered containing the calculated days for auto cleaning
  • Introduce a AdministrationConfig database table like already exist for scheduler : SchedulerConfig (adopt behavior, just for administration domain)
  • Enhance AdministrationConfig and introduce time field auto_cleanup_in_days - per default 0, means auto cleanup disabled

UC: Administrator fetches auto cleanup configuration

Domain: administration

  • REST API for admin domain
  • Open API generation check
  • When no cleanup configuration existed before, a default will be returned with "0 month" - so still being disabled

UC: Auto clean execution by system (technical use case)

Domain: scan

  • Rename old ScanConfig to ScanMappingConfig and all related parts (like services etc.) - will be done with #1059
  • Introduce new ScanConfig and adopt like done inside introduce time field auto_cleanup_in_days - per default 0, means auto cleanup disabled
  • on AUTO_CLEANUP_CONFIGURATION_CHANGED the days for cleanup are fetched from event and stored in db
  • Introduce ScanAutoCleanupTriggerService which is annotated with @schedule and is configurable by SpringValue annotion
  • delete old scan_product_result entries
  • delete old scan_project_log entries
  • we currently do NOT delete scan_report entries at the moment, this will be done with #1010 because currently we can only use these database rows for statistics

Domain: schedule

  • on AUTO_CLEANUP_CONFIGURATION_CHANGED the days for cleanup are fetched from event and stored in db
  • Introduce ScheduleAutoCleanupTriggerService which is annotated with @schedule and is configurable by SpringValue annotion
  • drops old schedule_sechub_job entries

Domain: administration

  • on AUTO_CLEANUP_CONFIGURATION_CHANGED the days for cleanup are fetched from event and stored in db
  • Introduce a AdministrationAutoCleanupConfig which represents the auto cleanup configuration
    This configuration contains a "remove-jobs-older-than" field which contains a "time-unit" and "amount" entry.
  • Introduce AdministrationAutoCleanupTriggerService which is annotated with @schedule and is configurable by SpringValue annotion. Does use the AdministrationConfigService to fetch data
  • AdministrationAutoCleanupService : deletes old adm_job_information entries

Documentation

  • Rest doc
  • Use case documentation are descriptive (+ don't forget the technical use cases)
  • Short explanation of concept "auto cleanup" inside "concepts"
  • The format of the time unit can be defined in "days", "weeks", "month", "years". This must be documented

Integration test

  • setup an integration test (difficult but should be possible)

Automatically remove old security scan result in products

With #19 sechub will automatically remove old scan results stored in SecHub database.
Out of scope there is the delete of scan data inside security products itself.

If this issue is done/necessary, we have to do following:

  • introduce a new event which has to be triggered by Job implemented in #19
  • implement a service which is able to read the product results, get corresponding data for product
    itself from result and trigger product adapter to delete
  • enhance adapter interface and implementations to provide the delete

Provide authenticated login for webscans

At the moment web scans are only done unauthenticated.
This reduces possibility of finding vulnerabilities much.

Current config (anonymous only)

{

	"apiVersion": "1.0",

	"webScan"   : {
               "uris": ["https://productfailure.demo.example.org"]
         }
}

Provided login mechanism

Some web scanners provide different authentication methods:

  • basic
  • form based
    • automated
    • scripted
    • recorded

We want to provide all of them without being product specific.
Basic and form based automated are simple - for "scripting or "recorded" this will be tricky.

Handle basic login

just provide product with information

Handle form based automated login

just provide product with information

Handle recorded or scripted logins

"Recorded" is a little bit "evil" when wanting to be product independent. Scripting as well. To solve this we will provide a scripting part where operations can be described in a very simple way. A product dependent script will be generated. If the product needs a recorded script/file, we will generate the record entry... All necessary metadata should be available when defining necessary script steps inside configuration.

Provide Web UI for administrators, users and owners

Currently we provide only a spartanic admin ui for developers.

We must provide a web admin UI for better and comfortable administration. The developer tools admin ui is only a first step or something to debug / test features etc. but not really a tool for operation.

Wanted:

  • server side rendered
  • shall be autark deployable
    (so can be outside normal sechub server url => not an easy target...)
  • ... more

Provide possibility to terminate SecHub Jobs

As a sechub project user I want to terminate a trigger sechub job.

  • when not started
  • when already running

This shall be possible by

  • REST API
  • go client

Result shall be that the product adapters terminate the running product instances as well - if possible.

Online documentation update

TODO

Do read (complete)

  • techdoc
  • architecture
  • rest api
  • client documentation

and fix typos, add missing parts etc.

Improve sechub configuration validation

Sechub configuration validation is not very readable.

Also we need following additional validations:

  • infrascans must contain at least one IP or one URI
  • webscans must contain at least one URI

Provide possiblity to protect/differ aministrative and normal REST API by profiles

Why this feature ?

REST calls starting with /api/admin should be better protectable by firewall / network configuration.

We provide a new profile "ADMIN_ACCESS" and make all admin rest parts only available by this profile. So we can start server with or without access to administrative parts.

Benefits:

  • for integration tests we can enable ADMIN_ACCESS as well, to have got only one part and one server starting
  • for production we are able to start different servers: some/one for user access and another one for admin access only which can be restricted by firewalls - e.g. port and ip access filtering
  • no special configuration or additional software necessary

Document branching model

Direct committing to master branch is not okay.

This must be documented and an (easy) branching model defined

Reduce boiler plate code for Messaging

At the current situation we must duplicate always the information about sending/receiving a message in annoatation and the real doing inside code with same message id.

This is very cumbersome:

In next example we got REQUEST_SCHEDULER_STATUS_UPDATE always mentioned twice

@IsSendingAsyncMessage(MessageID.REQUEST_SCHEDULER_STATUS_UPDATE)
	private void sendUpdateSchedulerStatusEvent() {
		DomainMessage request = DomainMessageFactory.createEmptyRequest(MessageID.REQUEST_SCHEDULER_STATUS_UPDATE);
		eventBusService.sendAsynchron(request);
	}

Variant 1

If would be nice to have something like
DomainMessageFactory.createEmptyRequest(DomainMessageUtil.getMethodMessageId());

or another solution where we do not have to duplicate the message ids.

Additional thought: One benefit of the duplication is the possiblity to mark another (upper) method for the documentation but handle on another - this would be no longer possible than.

Eat your own dogfood: Static code analysis SecHub OSS inside CI

inside continous-integration-multibranch-pipeline.jenkins there shall be another stage for security code scanning.

We need following value defined in JENKINS
SECHUB_CODESCAN_ENABLED (default:false, than scan shall be ignored)
SECHUB_SERVER - shall be used for parameter server

We need following secure storage parts:
SECHUB_APITOKEN - the token
SECHUB_USERID shall be used for parameter userid

project name will always be sechub

Concept: False positive handling

We will create an asciidoc file for the concept
sechub-doc/src/docs/asciidoc/documents/shared/concepts/false-postitive-handling/false-postitive-handling.adoc

At the moment false positive handling must be done inside the security products

But what happens when somebody switches products or does use two products for same purpose ?
=> You got false positives again.

Code scan

Mark false positives in code by developers

One thing how this could be handled very smooth by developers could be a single line comment option before the false positive
// NOSECHUB:${identifier}
E.g.
Having a medium finding like:

"description": "\n<br>Location:java/com/daimler/sechub/docgen/AsciidocGenerator.java - line:28, column:35\\n<br>For details...
"name": "Absolute Path Traversal",

A developer could add a comment before ala
// NOSECHUB:Absolute Path Traversal

SecHub would let use scan results from products, but filter the corresponding findings, by mapping with the found NOSECHUB lines.

This has the benefit, that when developers refactor their code and the method comes up at another position or even at another file the false/positive handling would be still working!

Web scan

  1. REST service with filter option for URL + Vulnerability
  2. later: Web UI - using 1.

Infra scan

TBD

Excel export for developertools admin UI

Developertools admin ui shall have an excel export for user/owner/project overview:

We need a simple export to excel were we can find

  • projects, with
    • their name, owner, owner email address, assigned users, whitelists

Online documentation bugfix

At this issue we collect current bugs in online documentation (github pages)

By this issues the problems will be fixed inside adoc files.
When fixed we generate new html files and push them to repository - but
we do no micro pushing...

Found problems

  • Description of System Properites does not describe that you have the possibility to use also ENV entries for this and how the conversion is done (standard spring way, but should be mentioned inside documentation)
  • download parts in github pages must be removed, because gives no benefit
  • testmode must be removed
  • Fix "RestController delegates to RestController" in techdoc
  • Document new Gradle tasks
  • include messaging parts into techdoc (currently only available at architecture)

Publish changes

  • Build new github pages and push

Prod deployment not working, because SSL certificate missing

PROD deployment failed because SSL certificate for PROD not found/available,
output was:

WARN 7 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is java.lang.IllegalArgumentException: Resource location must not be null

Provide SecHub scan profiles in sechub.json

User story:

... "As a SecHub user I want to use a dedicated profile in my sechub.json file to have the possibiility to change the security tools behaviour"

Description:
We need profiles to handle specific security scenarios/needs.

E.g. a scenario like "continous-integration" shall check faster/simpler than a "check-before-release" .

Technical:

  • provide profile-ids inside database
  • check profiles in json valid
  • provide a mapping from profile-id (for user) to internal used profile in products
  • use profile part in adapter communication

Make product executors configurable at runtime

At the moment SecHub does provide only 3 products in a static way.

Problems:

  • All three products must be configured /prepareed
    (otherwise server start fails)
  • No dynamic configuration of used products
    • No "kill switch" for products
    • No "activate" for short times /evaluation etc.

Wanted solution:

  • Configuration inside database
  • Rest API to enable/disable (and configure?) products

Integrate github actions

  • Connect to travis CI
    • shall do all parts from jenkins CI pipeline
    • represents a quick CI visible for community, Jenkins still necessary for own builds
  • Link images from travis builds to README.md in root folder

Change initial server and client tags

We tagged initial OSS server with version 1.0.4 and client with 1.5.0, because this were the versions used inside the former internal project.

But having multiple changes in API in next future (downward compatible but still changes) and still wanting to support only API v1.0 at the moment and wanting to show that sechub is still in development we decided to change back the initial versions:

rename v1.0.4-server -> v0.10.4-server
rename v1.5.0-client -> v0.15.0-client

SecHub is ready to use with 3 products only currently. It works, but there is still much to do!
So this will represent more the current situation

So inside this issue, we will remove the old tags and replace them with new ones

Provide and describe possibility to use SecHub behind a SSL proxy

Its not recommended, but if anybody wants to use SecHub with HTTP only (e.g. when running behind a SSL proxy) this shall be provided and also described in documentation.

Tasks:

  • Describe in documentation how to setup server for running only HTTP
  • Make protocol changeable in developer administration UI

Create a ready to use java library for REST access

We got a go client which does read the sechub.json file and does all stuff.

But at some situations - e.g. when using SecHub from another software system - using REST API is necessary instead of using the native client.

Target:
For java we could simply provide a nice, small and convenient java library for automation.

To think about
How should we do the rest calls here? Should we use spring boot (so not small any more...) or maybe just provide a helper class to create URLs easier and some POJOs and the communication could be done by callers themself.

Write operations documentation

Write a documentation for people only want to know how to operate SecHub as server in their existing infrastructure.

Former documentation does only handle developers, "users and architecture.

Provide OWASP ZAP

Situation

At the moment we only provide one SAST component: netsparker

Wanted

We want to provide OWASP ZAP as SAST component (FOSS alternative) into SecHub. See
https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project

Solution

  • We integrate Owasp zap by PDS approach.

  • We use this issue as an epic until we have

    • Sereco and Report Support for OWASP
    • Full PDS integration
    • Full login support
  • Work is done in "sub issues" linked inside this issue.

HTML report has wrong title name

When some body wants to save the html report generated by sechub, the name will be
SecHub scan result"

So we got a double quote at the end.

This is a typo and also a problem when moving / copying files at linux.

Must be fixed

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.