regnology / lucy

Lucy is a component analysis platform to minimize the risk of license infringements and to support and optimize the license compliance process.

Lucy
Licenses under control, yeah!

Lucy is a component analysis platform to minimize the risk of license infringements and to support and optimize the license compliance process.

Third-party and open-source software typically has one or more licenses assigned. The chosen license may or may not allow certain types of usage, contain distribution requirements or limitations, or require specific actions if the component is modified. Lucy can identify the license(s) for a given component and provide guidance as to the nature of the license (i.e. license text, copyright, risk, source code, etc). Utilizing components with licenses which conflict with an organizations objectives or ability can create serious risk to the business.

Features

• Track usage of components in projects
• Manage licenses with risk and requirements
• Supports standardized SPDX ID's
• Supports different uploads:
  • CycloneDX SBOM (XML)
  • CSV
  • Archives with Maven components (JAR, ZIP, WAR, etc.)
• Ecosystem agnostic with built-in repository support such as autocomplete of missing information for:
  • Maven (Java)
  • npm (Javascript)
  • PyPI (Python)
  • NuGet (.NET)
  • Golang (Go)
• Generate various reports to ensure compliant software deliveries
• Mapping for different license names
• Detects incompatible license combinations
• Supports Fossology API to start analysis of components
• Metrics for projects
• Monitor the development of projects and compare releases
• API-first design
• Local manageable users
• Runs as Docker container

Build Docker Image

Lucy requires at least Java version 11.

Currently, only the correct and full functionality of Lucy is supported as a Docker image.

Build Docker image:

./mvnw package -Pprod -DskipTests verify jib:dockerBuild

Lucy is based on the JHipster framework. For further information check JHipster.

Usage

The image is launched using docker-compose. Under src/main/docker/app.yml is a sample file. This starts Lucy and a PostgreSQL. The docker-compose file must be adapted to the corresponding infrastructure afterwards.

Start Lucy image:

docker-compose up -d

Lucy is reachable at localhost:8090 by default.

Quickstart

When using Lucy for the first time, the user "admin" with password "admin" and "user" with password "user" exist. The passwords should be changed!

Likewise, other users can be created. The following roles can be assigned to users:

Role	Description
Admin	Full rights and access to admin view
User	Rights to view, create, edit, delete products, components, licenses, etc.
Readonly	Rights to view products, components, licenses, etc.

To start the initial analysis of an SBOM, it is recommended to create licenses first. Because without licenses the components cannot be identified correctly.

For a list of existing licenses, the SPDX License List can be used. Each license should be assigned a risk and requirements. Risks and requirements are already predefined.

For the first analysis, a new product can now be created. In the product, an SBOM can be uploaded using the "Upload" button. It is recommended to create an SBOM with CycloneDX. Only the XML format is supported.

Depending on the size of the SBOM, the first upload may take a while. After successful processing, the components are listed in Lucy.

Main Page	Product Detail Page

Copyright & License

Lucy is Copyright (c) 2023 Regnology GmbH. All rights reserved.

Lucy has been developed to provide support in license handling, which by nature concerns legal matters, and it has been released under an MIT license to make it available to everyone interested in using it. To make the usage easier, the authors have included sample data and some basic configuration.

However, the examples provided in the database of the GitHub project are for reference only and should not be used as final or definitive solutions. These examples are not intended to provide legal advice and should not be relied upon as such. Users are advised to evaluate and verify the information and examples provided before taking any action based on them.

Please see the full MIT license text.

lucy's Issues

Rename "Library" to "Component"

Is your feature request related to a problem? Please describe.
The term "Library" should be renamed to "Component" everywhere in the front end of Lucy.

Describe the solution you'd like
"Component" is the more general name, we mostly use it instead of "Library"
The name "Library" would still appear in the DB, it would only be replaced in the Frontend

Describe alternatives you've considered
N/A

Additional context
In the back end, this is not possible because "Component" is a keyword in JHipster.

Add Copy Feature for Licenses from Another Library

Is your feature request related to a problem? Please describe.
Between two releases of the same product:

• Usually, we have similar libraries with different versions, which are under the same licenses.
• Currently, we need to enter the same licenses from scratch for every OSS List/ release.

Describe the solution you'd like
For every library, we add a button "copy/import" at “details”; where we can choose what library’s licenses can be copied

Features and Error handling

• A library that does not exist should either not be possible to be specified to copy from, or the result should be an empty set of licenses for the new library (ie. "do nothing")
• If the new library already has licenses added to it, it needs to be decided if the copied licenses should be appended, or overwrite the existing ones
• Ideally the user should decide the behaviour, either via a setting or as an interactive menu
• Depending on the implementation, a mass copy function would be helpful, which copies a large(r) amount of licenses (if that is possible without too much work)
• The source library (to copy from) should not be identical to the target library, or alternatively, in that case, make sure to do nothing

Acceptance criteria

• Function is implemented to copy licenses to a given library from another (existing) library
• Function is located at a place where it integrates well into the workflow
• Test: The function needs to work with different scenarios as specified under "Features and Error handling". Test if all specified scenarios work.

Describe alternatives you've considered
N/A

Additional context
Some libraries can have 10-15 licenses per library, so it will be helpful to have a copy/import function in Lucy for licenses (for every library). This way we can reduce manual work and save a lot of time.

Unknown license bug

Describe the bug
Sometimes the problem seems to occur that the "License To Publish" is Unknown, although licenses are set
The issue seems to occur when uploading archives (ZIP, JAR, ...)

To Reproduce
Steps to reproduce the behaviour:

1. Go to '/product/[id]/view'
2. Click on 'Upload'
3. Choose File (Upload a BOM (CSV, XML) or archive file format (ZIP, TAR, WAR..))
4. Click 'Upload'
5. After the Upload process is finished some entries have the "License To Publish" as Unknown

Expected behaviour
There should be no entries that have "License To Publish" as Unknown

Additional context
The bug is most likely related to the Upload Archive Memory bug #13 because this only happens when uploading archives and not with the regular csv or xml BOM uploads

Invalid License URL from NPM registry breaks BOM upload "invalid URI scheme"

Describe the bug
When uploading a bom file for a product, if the URL received from the NPM registry for downloading the license text is wrong, then the upload process stops and breaks.

To Reproduce
Steps to reproduce the behavior:

1. Upload a bom file that contains btoa version 1.2.1 as a component;
   <component type="library" bom-ref="pkg:npm/[email protected]"> <name>btoa</name> <version>1.2.1</version> <purl>pkg:npm/[email protected]</purl> <properties> <property name="aquasecurity:trivy:PkgID">[email protected]</property> <property name="aquasecurity:trivy:PkgType">npm</property> </properties> </component>
2. Because the registry is returning the following response that contains a wrong URL.
   repository":{"type":"git","url":"git://git.coolaj86.com/coolaj86/btoa.js.git"}
3. The function libraryService.urlAutocomplete(dbLibrary); is breaking when trying to download the License and Source code
4. See error invalid URI scheme git

Expected behavior
When encountering invalid URLs Lucy should discard them instead of trying to make the request

Fossology scan bug

Describe the bug
When scanning a component with Fossology via Lucy, a problem occurs if the component name contains slashes. This is because the component is stored and made available locally for Fossology to download.

Since the name contains slashes, this is interpreted as subdirectories and thus the file cannot be found where it is expected.

To Reproduce
Steps to reproduce the behavior:

1. Create a library with slashes (/) in the artifact ID
2. Start Scan with Fossology

Expected behavior
Start scan with Fossology and find the source code archive in the filesArchive/fossology

Screenshots

Additional context
Replace slashes with another character like "_", when making it available to download

License risk bug

Background:
Sometimes the license risk is missing. This is happening when we have an unknown library and I fix it with the correct license.

The correct license is saved but the license risk doesn't appear.

The problem fixes itself when I click on "edit", after click "save".

To reconstruct the problem, please follow the steps:

• Open any library and select the "unknown" license. Click save.
• Open the same library and select any license. Click save.
• For solving the problem- click on "edit" then "save" without modifying anything.

Test feature

This is a new feature!

Upload Archive Memory bug

Describe the bug
When doing an upload of an archive the RAM that is needed to analyse the archive doesn't get freed after the analysis has finished.

To Reproduce
Steps to reproduce the behaviour:

1. Go to '/product/[id]/view'
2. Click on 'Upload'
3. Choose File (Upload a BOM (CSV, XML) or archive file format (ZIP, TAR, WAR..))
4. Click 'Upload'
5. After the Upload process is finished the memory is not freed

Expected behaviour
After the Upload process is finished the memory should be freed

Additional context
The classes for processing an archive were mostly rewritten and optimized. The old implementation contained a few errors. The rewrite has partially improved the memory problems, but not completely.

Also, the problem was that previously several different uploads were mixed together, which is not allowed to happen under any circumstances. I have not tested this again until now to see if this is now fixed.

Uploads of archives should therefore still be treated with caution and after each upload, it is best to restart the container to free up memory.

The uploaded archive still takes up memory space after the upload has been processed. It's less than before but still not good enough.

A deeper analysis of the garbage collector needs to be done.

Error log bug

When we have a library under two (or more) incompatible licenses, Lucy should trigger an error log and we should be able to see it in the overview page.
I checked and it seems to be a bug. The right information is saved in the database, but not available in the error log details.

Test Issue

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

This is a test issue

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Option to change the number of data entries per page

Is your feature request related to a problem? Please describe.
Right now we show 50 entries per page by default, meaning that every view that has a list of entries, queries only 50 entries at once.

Describe the solution you'd like
To make it more user-friendly, we should add an option to change this in the front end using a dropdown box with the options to display 20, 50, 100, or 500 entries per page.

Describe alternatives you've considered
For some pages (views) it makes more sense to use endless scrolling (continuous loading when scrolling down)
A show all button can also be considered

Additional context
N/A

Add additional search parameters for library overview.

Is your feature request related to a problem? Please describe.
Improve the library overview. We need new options for the search bar to search and filter by more fields.

Describe the solution you'd like

Front end:

Search bar

• Add new fields and toggle buttons for search / filters
• Add new attributes to the searchForm in the components.ts file

Back end:

• Add or adjust fields in the existing filter class if necessary

Describe alternatives you've considered
N/A

Additional context
We need new options for the search bar to search and filter by more fields.

Conflicting Licenses Logic

Is your feature request related to a problem? Please describe.
In some components having multiple licenses, there might be a conflict between the licenses when copylefted licenses meet a permissive license which protects itself again copyleft takeover. Lucy should be able to show these license conflicts.

Describe the solution you'd like

Front end:

• New view to edit, view the list with the conflicting licenses or add new fields for the license edit view to enter the conflicting licenses
• Show conflicting licenses in the license detail view
• (Optional) Upload of a CSV file with the conflicting licenses. Requires a fixed format that must certainly always be followed
• Show conflicting licenses for libraries in the product detail view

Back end:

• Add a new table to save license conflicts in the database. One license can have multiple license conflicts and one license conflict belongs to multiple licenses (n:m)

Describe alternatives you've considered
N/A

Additional context
This feature is partly implemented.

For every license, the license compatibility with another license is visible in Lucy. To see them you have to go to a license and click on the tab "License Conflicts (BETA)".
By editing a license the compatibility to every license can be defined:

• Compatible
• Incompatible
• Unknown

Currently, the method which checks every library if it contains an incompatible license combination and creates an error log is disabled because the current implementation was not fast enough.

The solution would be to cache the license combinations and don't pull them for every check from the database.

Also, it is not necessary to cross-check every license with another but just to check every license with its next neighbour. That should also increase the performance.

Update License year to 2023

Is your feature request related to a problem? Please describe.
Update License year to 2023

Add dropdown menu for library overview.

Is your feature request related to a problem? Please describe.
Add dropdown menu to the products page. This should allow for additional options to sort by like the one in license page.

Describe the solution you'd like

Front end:

Dropdown

• Add new fields and toggle buttons for showing additional information columns.
• Add ability to sort/order by new columns.

Back end:

• Add or adjust fields in the existing filter class if necessary

Describe alternatives you've considered
N/A

Additional context
We need the ability to better sort data on the product page.

Comment Field Bug in CSV Export

Describe the bug
When we export the full report from Lucy, the comment field is wrongly exported, mingled with libraries.

To Reproduce
Steps to reproduce the behavior:
1.Open a product
2. Go to "Reports"
3. Click on OSS List > Full > Download
4. See - the field "Compliance Comment" mingled with libraries.

Expected behavior
One solution may be to put the compliance comment into " "-> which is the usual text identification in CSV (for Excel).
It might make sense to put all text fields in ""
Downloaded CSV with large comments can be opened in Excel without breaking the format of the table.

Show Fossology results in Lucy

Is your feature request related to a problem? Please describe.
Currently, the results of the Fossology scan can only be viewed directly in Fossology.

Describe the solution you'd like
It would therefore be helpful if we could also see the results directly in Lucy. In addition, it should be possible to transfer these results to the library license field.

With the following API request, it is possible to get the result of an upload:

{{curl -X 'GET' }}
{{ 'http://localhost/repo/api/v1/uploads/{ID}/licenses?agent=nomos,monk,ninka,ojo,reportImport,reso,scancode&containers=true' }}
-H 'accept: application/json'

After the upload, a background task is running and we will know when the scan is done (Progress bar).

Considerations

• the results have to be reviewed manually (a license from within a component doesn´t have to apply for the component fully)
• this requires maybe a temporary / review table before saving results into the DB

Describe alternatives you've considered
N/A

Additional context
N/A

Include "Type" to unique Library identifier

Is your feature request related to a problem? Please describe.
Right now only GroupId, ArtifactId, and Version are used as unique identifiers. Without Type, we can have duplications (especially during uploads). This may lead to problems the larger the database gets. This is to distinguish the type "jar" and "py" or other files.

Describe the solution you'd like
The "type" information is included in the BOM files (CycloneDX). In some cases, it may be missing though (e.g. Tern where it should be "Docker").

Describe alternatives you've considered
N/A

Additional context
A decision needs to be made on how to handle the data coming from the BOM file in case the "type" is not available for some reason.