sw360 / sw360chores Goto Github PK

I got the latest sources from sw360chores and followed the instructions of V2 to build everything.
That works fine, however when I run startUp.sh and go to https://localhost:8443 I get an Bad Gateway error with the following Nginx error.

I'm running this on a VirtualBox and all docker containers are running

Did I miss something? Any help will very much appreciated.

Followed procedure: https://gist.github.com/roycyt/b6ddd241caf734732d1780d5de27a6ec

using latest stable release: sw360-8.1.0-M1

In the fresh install of sw360 via docker chores

We have the sw360-8.1.0-M1

I have imported users as well

and tried to login with different users as well (admin, user etc)

Error log:

Components_ErrorLog.txt

Highlights of error

 ERROR [http-nio-8080-exec-15][ComponentPortlet:1451] Problem retrieving all the Vendor names
 org.apache.thrift.transport.TTransportException: HTTP Response code: 404

ERROR [http-nio-8080-exec-15][UserCacheHolder:75] Unable to fetch user...
 java.util.concurrent.ExecutionException: org.apache.thrift.transport.TTransportException: HTTP Response code: 404


ERROR [http-nio-8080-exec-15][ComponentPortlet:1475] Could not get component total count in backend
org.apache.thrift.transport.TTransportException: HTTP Response code: 404


ERROR [http-nio-8080-exec-15][PortletRequestDispatcherImpl:303] Unable to dispatch request: java.lang.InstantiationException: bean totalRows not found within scope
ERROR [http-nio-8080-exec-15][PortletServlet:119] javax.portlet.PortletException: javax.servlet.ServletException: java.lang.InstantiationException: bean totalRows not found within scope
javax.portlet.PortletException: javax.servlet.ServletException: java.lang.InstantiationException: bean totalRows not found within scope


ERROR [http-nio-8080-exec-15][render_portlet_jsp:131] null
 java.lang.InstantiationException: bean totalRows not found within scope

Also refer https://sw360chat.slack.com/archives/C68C2RYS3/p1586447237060000

In deployment, the port to the thrift services may be closed (or configured to access only from localhost only) - it is open by default.

When deploying using the sw360dependencies.tar.gz build by the prepare script, the couchdb-lucene web app is created with suffix -1.0.2.war which is also in the resulting URL of this web app.

The problem is that sw360 / the couchdb config does not expect the number and thus ... workaround is to manually renamed the couchdb-lucene-x.y.z.war to couchdb-lucene.war

Goal: We should allow windows users to easily setup a sw360 test instance to develop against, e.g. the sw360 rest api

1. should be self contained. Pull latest version of sw360chores from github
2. creates all images from sw360chores
3. should build sw360portal
4. should deploy sw360portal wars into tomcat container

The sw360.properties might change at runtime, should be externalized as the fossology conf (as single big chunk of key-values).

Attempting to build on osx results in build error for couchdb-lucene

./sw360chores.pl --build -- up
  variables:
    $projectName  = sw360
    $build        = 1
    $save         =
    $pushTo       =
    $cleanup      =
    $prod         = 0
    $swarm        = 0
    $cveSearch    = 0
    $watchtower   = 0
    $cpWebappsDir = <undefined>
    $cpDeployDir  = <undefined>
    $backupDir    =
    $restoreDir   =
    $debug        = 1
  environmental variables:
  arguments:
    @ARGV         = up

[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:15 min
[INFO] Finished at: 2019-10-15T20:21:10Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:2.6:resources (default-resources) on project couchdb-lucene: Cannot create resource output directory: /couchdb-lucene/target/classes -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
failed to prepare couchdb-lucene at ./sw360chores.pl line 380.

We need to add releases to this repository.

Maybe we should also add a disclaimer like

Release Pattern

This project does not follow semantic versioning - version numbers are kept matching to the minimum compatible product version

it seems like on macosx (10.13.6), docker setup is not running:

lando:sw360chores-master sam$ docker -v
Docker version 18.06.1-ce, build e68fc7a
lando:sw360chores-master sam$ ./sw360chores.pl --build -- up
... the file liferay-portal-tomcat-6.2-ce-ga5-20151119152357409.zip already exists: skip
rm: /Users/sam/Downloads/sw360chores-master/_tmp/tmp.Dw9kEAC/sw360/webapps/docs: No such file or directory
rm: /Users/sam/Downloads/sw360chores-master/_tmp/tmp.Dw9kEAC/sw360/webapps/examples: No such file or directory
rm: /Users/sam/Downloads/sw360chores-master/_tmp/tmp.Dw9kEAC/sw360/webapps/host-manager: No such file or directory
rm: /Users/sam/Downloads/sw360chores-master/_tmp/tmp.Dw9kEAC/sw360/webapps/manager: No such file or directory
sed: 1: "/Users/sam/Downloads/sw ...": bad flag in substitute command: 's'
failed to prepare sw360empty at ./sw360chores.pl line 311.
lando:sw360chores-master sam$ 

which maybe a platform issue because of different paths in the prepare scripts.

cve-search-server/Dockerfile contains a hardcoded commit hash for the cve-search project:
ENV BRANCH=4c165eff1af0e4c7bdf103c341203717ae677f64

The population of the cve database fails because NVD have retired the XML feed of CVE records.
https://nvd.nist.gov/General/News/XML-Vulnerability-Feed-Retirement-Phase-3

Version 2.3 of cve-search (Sep 18, 2019) https://github.com/cve-search/cve-search/releases/tag/v2.3 adds support for the new JSON format from NVD.

I am in a corporate network with a quite rigid proxy that requires authentication. Therefore, I use a local proxy like cntlm to have a rather transparent internet connection. In such an environment, the build of the Docker images does not work, even if the proxy settings (pointing to localhost in this case) are specified in the proxy.env configuration file; the containers cannot connect to the proxy.

A solution that worked for me was enabling the network mode 'host' when building images; it would therefore be nice if the main script could support a parameter to enable this mode and pass the corresponding arguments through to the Docker builds.

for a simpler deployment it would be good if we could create prepopulated docker containers.

Hello all,

I want to run SW360 together with Fossology and CVE search is this possible with the existing docker image?
Where should the Fossology configuration be done? - I assume in the configuration.yml but in the wiki entry the /backend/src/src-fossology/src/main/resources/fossology.properties file is referenced. Furthermore, my SW360 portal tabs (Components, Licences, ECC, etc.) are empty - should they all be empty as long as no connection with Fossology is established?

What I have already done:

• Fossology running in an own docker container
• the perl script for the sw360chores started which runs also docker containers with the described components.
• followed the Liferay deploy wiki entry
• started following the Fossology deploy wiki entry

Thanks in advance!

From @bs-jokri on October 9, 2017 11:1

From @maxhbr on June 19, 2017 11:4

using the current sw360/sw360rest@6ce17c1 one can integrate its docker-compose setup into sw360chores via the following changes:

1. name the images created by sw360rest

diff --git a/docker-compose.yml b/docker-compose.yml
index 65d8271..a0b6f1b 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -15,6 +15,7 @@ services:
 
   sw360-authorization-server:
     build: subprojects/authorization-server
+    image: sw360restauthorization
     ports:
       - 8090:8090
     environment:
@@ -22,6 +23,7 @@ services:
 
   sw360-resource-server:
     build: subprojects/resource-server
+    image: sw360restresource
     ports:
       - 8091:8091
     links:

(should be moved to upstream)

2. build the named images (in sw360rest):

$ ./gradlew build preparedocker
$ docker-compose build

3. include the new images to the sw360chores docker-compose setup

diff --git a/deployment/configuration.env b/deployment/configuration.env
index b757d81..37e6c7e 100644
--- a/deployment/configuration.env
+++ b/deployment/configuration.env
@@ -7,6 +7,7 @@
 # http://www.eclipse.org/legal/epl-v10.html
 
 DEV_MODE=true
+REST=true
 # CVE_SEARCH=false
 # HTTPS_COUCHDB=false
 # BACKUP_FOLDER=./_backup
diff --git a/deployment/docker-compose.rest.yml b/deployment/docker-compose.rest.yml
new file mode 100644
index 0000000..d3b799b
--- /dev/null
+++ b/deployment/docker-compose.rest.yml
@@ -0,0 +1,30 @@
+# Copyright Bosch Software Innovations GmbH, 2017.
+# Part of the SW360 Portal Project.
+#
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Eclipse Public License v1.0
+# which accompanies this distribution, and is available at
+# http://www.eclipse.org/legal/epl-v10.html
+
+
+# This file adds a cve-search server to the sw360 environment.
+# Use it together with the main docker-compose.yml as described
+# in README.md
+
+version: '2'
+services:
+  sw360-authorization-server:
+    image: sw360restauthorization
+    ports:
+      - 8090:8090
+    environment:
+      - SW360_AUTHORIZATION_SERVER_ARGS=-Dspring.profiles.active=dev
+
+  sw360-resource-server:
+    image: sw360restresource
+    ports:
+      - 8091:8091
+    networks:
+      - sw360front
+    environment:
+      - SW360_THRIFT_SERVER_URL=http://sw360:8080
diff --git a/deployment/docker-compose.sh b/deployment/docker-compose.sh
index 3fcac89..5b718f4 100755
--- a/deployment/docker-compose.sh
+++ b/deployment/docker-compose.sh
@@ -92,6 +92,7 @@ addSudoIfNeeded() {
 cmdDocker="$(addSudoIfNeeded) env $(grep -v '^#' proxy.env | xargs) docker"
 cmdDockerCompose="${cmdDocker}-compose -f $DIR/docker-compose.yml"
 [ "$DEV_MODE" == "true" ] && cmdDockerCompose="$cmdDockerCompose -f $DIR/docker-compose.dev.yml"
+[ "$REST" == "true" ] && cmdDockerCompose="$cmdDockerCompose -f $DIR/docker-compose.rest.yml"
 [ "$CVE_SEARCH" == "true" ] && {
     cmdDockerCompose="$cmdDockerCompose -f $DIR/docker-compose.cve-search-server.yml"
     [ "$DEV_MODE" == "true" ] && cmdDockerCompose="$cmdDockerCompose -f $DIR/docker-compose.dev.cve-search-server.yml"

(based on https://github.com/bsinno/sw360-chores/tree/integration/2017_05_31)

Copied from original issue: bsinno/sw360#442

Copied from original issue: sw360/sw360portal#649

In the script /deployment/couchdb-lucene/prepare.sh the proxy is not configured correctly. mvn does not use the environment for proxy settings.

Currently a lot of traffic is created when loading the sw360 pages. Some investigation reveals that a lot of small css other files are fetched by the browser.

In order to improve the responsiveness of the page browser caching could help. This feature needs to be configured at the nginx web server.

See eg
https://www.digitalocean.com/community/tutorials/how-to-implement-browser-caching-with-nginx-s-header-module-on-centos-7

for details.

From @bs-jokri on September 7, 2017 14:43

Problem is, that the links in the ui are not porperly created if another port is used.

Copied from original issue: bsinno/sw360#484

Currently, connection is refused, which is confusing for most users, since the nginx could easily redirect the traffic arriving at 80 to 443.

The tomcatdeploy.sh script monitors a directory for new or updated web applications to be deployed. When a change in this directory is detected, the file affected is moved into the actual deployment folder of Tomcat (lines 40 and following).

When trying to setup sw360chores, I had the situation that the directory to monitor did not exist yet. This caused the monitoring mechanism to fail, and no deployment was done.

A fix would be to make sure that directory always exists, creating it if necessary.

docker-images/sw360empty/docker-entrypoint.sh

if [ ! "$POSTGRES_HOST" ] || [ ! "$POSTGRES_USER" ] || ( [ ! -f "$POSTGRES_PASSWORD_FILE" ] && [ ! "$POSTGRES_PASSWORD"] ); then
    echo "postgres configuration incomplete"
    exit 1
fi

should be [ ! "$POSTGRES_PASSWORD" ] otherwise issues with script

according to [1] one has to have

• a sql dump of the postgres and
• the content of $CATALINA_HOME/data

to restore an liferay instance. We could use content, from a freshly provisioned instance, to create some files which can be used while building the images, to get an already provisioned liferay.

These two parts thus could be added as default content to sw360postgres and sw360empty respectively.

[1] https://web.liferay.com/de/web/user.26526/blog/-/blogs/backup-restore-a-liferay-environment

For the line:

The new link is:

https://github.com/danielebailo/couchdb-dump/

Hello,
i would like to try the docker deployment, I will test the pl script but do you consider creating a helm deployment for a kubernetes usage ?

it seem like executing the single line as given in the current readme.md:

$ mvn install -P deploy -Ddeploy.dir=/ABSOLUTE/PATH/TO/sw360chores/_deploy -DskipTests

is not working, because maven vields the following message:

[INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (enforce) @ sw360 ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireProperty failed with message:
You must set at least the property 'base.deploy.dir'!

rather the command could be (works for me):

mvn package -P deploy -Dbase.deploy.dir=/ABSOLUTE/PATH/TO/sw360chores/_deploy -Dliferay.deploy.dir=/ABSOLUTE/PATH/TO/sw360chores/_deploy/deploy -Dbackend.deploy.dir=/ABSOLUTE/PATH/TO/sw360chores/_deploy/tomcat-9.0.17/webapps -Drest.deploy.dir=/ABSOLUTE/PATH/TO/sw360chores/_deploy/tomcat-9.0.17/webapps -DskipTests

I began by creating a Rest API token for Fossology.

I followed the fossology documentation, and everything worked in the terminal: https://www.fossology.org/get-started/basic-rest-api-calls/

I have a folder with the id of 3, but there is an error when trying to access fossology from sw360.

The current chores setup lacks the recently required apache commons compress dependency from updating apache poi to four-something in this commit:

eclipse-sw360/sw360@71348b4

It needs to be added here:

https://github.com/sw360/sw360chores/blob/master/miscellaneous/prepare-liferay/prepare.sh

with https://repo1.maven.org/maven2/org/apache/commons/commons-compress/1.20/commons-compress-1.20.jar

(should be sufficient)

Hello
do you know if the docker images are available on a docker registry like docker hub ?

See corresponding issue eclipse-sw360/sw360#398.

After bringing up a deployment of sw360 via the following:

./sw360chores.pl --build -- up

Is it possible to configure or is there a default an admin account created I login with, instead of having to create (or sign up) an account post set up via the web UI?