spdx / tools-golang

Collection of Go packages to work with SPDX files

SPDX tools-golang

tools-golang is a collection of Go packages intended to make it easier for Go programs to work with SPDX® files.

Recent news

2022-01-11: v0.4.0: added support for SPDX v2.3 and YAML, as well as other improvements and bugfixes. See RELEASE-NOTES.md for full details.

What it does

tools-golang currently works with files conformant to versions 2.1 and 2.2 of the SPDX specification, available at: https://spdx.dev/specifications

tools-golang provides the following packages:

• spdx - in-memory data model for the sections of an SPDX document
• tagvalue - tag-value document reader and writer
• rdf - RDF document reader
• json - JSON document reader and writer
• yaml - YAML document reader and writer
• builder - builds "empty" SPDX document (with hashes) for directory contents
• idsearcher - searches for SPDX short-form IDs and builds an SPDX document
• licensediff - compares concluded licenses between files in two packages
• reporter - generates basic license count report from an SPDX document
• spdxlib - various utility functions for manipulating SPDX documents in memory
• utils - various utility functions that support the other tools-golang packages

Examples for how to use these packages can be found in the examples/ directory.

What it doesn't do

tools-golang doesn't currently support files under any version of the SPDX spec prior to v2.1

Documentation

SPDX tools-golang documentation is available on the pkg.go.dev website at https://pkg.go.dev/github.com/spdx/tools-golang.

Contributors

Thank you to all of the contributors to spdx/tools-golang. A full list can be found in the GitHub repo and in the release notes.

In particular, thank you to the following for major contributions:

JSON parsing and saving support was added by @specter25 as part of his Google Summer of Code 2021 project.

RDF parsing support was added by @RishabhBhatnagar as part of his Google Summer of Code 2020 project.

Licenses

As indicated in LICENSE-code, tools-golang source code files are provided and may be used, at your option, under either:

• Apache License, version 2.0 (Apache-2.0), OR
• GNU General Public License, version 2.0 or later (GPL-2.0-or-later).

As indicated in LICENSE-docs, tools-golang documentation files are provided and may be used under the Creative Commons Attribution 4.0 International license (CC-BY-4.0).

This README.md file is documentation:

SPDX-License-Identifier: CC-BY-4.0

Security

For security policy and reporting security issues, please refer to SECURITY.md

tools-golang's Issues

Fix multiline snippet copyright in tvsaver

Currently, in tvsaver, Snippet copyright text lines are not saved with a textify call:

tools-golang/tvsaver/saver2v1/save_snippet.go

Line 36 in feb8734

fmt.Fprintf(w, "SnippetCopyrightText: %s\n", sn.SnippetCopyrightText)

(compare, for Files,

tools-golang/tvsaver/saver2v1/save_file.go

Line 42 in feb8734

fmt.Fprintf(w, "FileCopyrightText: %s\n", textify(f.FileCopyrightText))

and corresponding test at

tools-golang/tvsaver/saver2v1/save_file_test.go

Line 213 in feb8734

func TestSaver2_1FileWrapsCopyrightMultiLine(t *testing.T) {

)

A textify call should likely be added in tvsaver for both 2.1 and 2.2, after confirming that the spec Snippet copyright lines should wrap if multi-lined.

Consider whether to add top-level map of all IDs

Separate tracking issue for @goneall's comment at #24 (comment):

This means that probably this should just be replaced with a top-level "UnpackagedFiles" map of ElementIDs to Files at the Document level, and pull it out of the "Packages" map altogether.

For the Java code, there is simply a map of all SpdxElements within a document which includes both packages and files (since they subclass elements). There is a separate property for the Package that references the files.

Consider whether to add this to the Document structure. It would enable quickly finding an element. In particular, for documents with multiple Packages and Files within each one, it would enable quickly finding a File by ID with a single lookup, rather than crawling through each Package's set of Files.

Fix build badges

The current build badge is still pointing to Travis. It should point to the results of the GitHub Actions build instead.

Add utility function for retrieving packages with arbitrary relationships

https://github.com/spdx/tools-golang/blob/main/spdxlib/described_elements.go

This file contains functions that allow you to retrieve the IDs of the packages described by the document.

Would it be possible to get a similar function that allows us to retrieve IDs of packages that have a given relationship, e.g. DEPENDS_ON/DEPENDENCY_OF, rather than specifically DESCRIBES/DESCRIBED_BY?

Something like:

func GetPackagesByRelationship(doc *spdx.Document2_2, relationship string, ref spdx.ElementID) ([]spdx.ElementID, error) { ... }

usage:
GetPackagesByRelationship(doc, "DEPENDENCY_OF", packageA)
GetPackagesByRelationship(doc, "DEPENDS_ON", packageB)
GetPackagesByRelationship(doc, "DESCRIBED_BY", document)

Failing Tests on Windows Machine

I am on a Windows 10 Home Version 1909 OS,
The tests are failing on my machine.

The following command was used to test all the packages recursively.
go test ./... was run in the root directory.

This link has the output of the test.

Does not follow spec

Just used this for the first time and I note that this code has 2 string slices in CreatorInfo called CreatorPersons and CreatorOrganizations.

However the spec for 2.2.1 only has a scalar Creator field which is specified in the NTIA report.

The spec is here: https://spdx.github.io/spdx-spec/v2-draft/ - hopefully I have this right

and the NTIA report: https://www.ntia.doc.gov/files/ntia/publications/ntia_sbom_formats_energy_brief_2021.pdf

builder: File paths are not relative

From at least one test when running idsearcher and builder on a directory, the file paths for FileName are being generated incorrectly as /filename.txt rather than ./filename.txt.

The SPDX spec requires that the FileName field consist of relative paths.

jsonloader: reorganize files

cc @specter25

In the json branch, the JSON parser files are all within a top-level directory called jsonloader2v2.

In order to maintain the same structure as the RDF and tag-value loaders, I'd suggest making the following changes:

• change the top-level directory name to jsonloader/
• create a subdirectory within that called parser2v2/
• keep the main jsonloader.go and jsonloader_test.go files within the top jsonloader/ directory
• move the other files into jsonloader/parser2v2/
• make corresponding edits to the jsonloader.go and example 8 files for the path changes

I hope this isn't overly complicated (or that it takes a lot of time). git mv should be very helpful here. Mostly I'm just asking if it can be restructured so that the pattern will be the same as the RDF and tag-value loader directories.

The main reason for this is because, although today there's only SPDX 2.2 that has JSON files, someday there will also be SPDX 3.0 -- and it would be better to be positioned for that now.

Rename primary branch to 'main'

see https://github.com/github/renaming#renaming-existing-branches

[tvloader] got unknown checksum type SHA512

Unable to parse the following SPDX file with tvloader package:

...
SPDXID: SPDXRef-File-34182904f819889b086a855ee5878f2a2e0b9037
FileChecksum: SHA1: 335029255dbadc3dfeb15c8e432078b1535b90de
FileChecksum: SHA256: 0852707ebbd7e78d71c1eea3fb7ac0348aa22124a8975f787c07217058e0fe20
FileChecksum: SHA512: 2bca030aeb4aff21505267c02230bbc91bea728070fe6b46089eb7c168deeb6e0d40195a307d8e39073f322cba7abaf81a1b92011ca249eba2b8d92aedc10e38
...

Got the following error:

WARN[0000] got unknown checksum type SHA512

Switch ExternalDocumentRefs to use SHA1 instead of SHA256

It appears that External Document References (see https://spdx.github.io/spdx-spec/2-document-creation-information/#26-external-document-references) may require SHA1 instead of SHA256 in SPDX 2.1 and 2.2.

To be confirmed, but if this is required, then tools-golang should be updated accordingly.

Relationship field in tag/value parser rejects valid values

Thanks for making this Go library! Having the ability to use SPDX files in Go is excellent.

I think I've found a bug in how the tag/value parser handles Relationship fields: SPDX 2.2 specifies this example is valid:

Relationship: SPDXRef-BobBrowser CONTAINS NOASSERTION

However, with a file containing

Relationship: SPDXRef-ion-tester CONTAINS NOASSERTION

the tvloader.Load2_2(r) call returns an error stating missing SPDXRef- prefix for element identifier.

The same file successfully passes with the Java validator:

$ java -jar /tmp/tools-java-0.0.6-jar-with-dependencies.jar Verify ./spdx-test-1.spdx TAG

This SPDX Document is valid.

I've made a reproduction repository at https://github.com/matthewkmayer/spdx-go-repro . It can be cloned then run with go run . to see the error.

I tried to find tests for this scenario in this repo but didn't find any. Perhaps guidance on how to find the relevant tests could help us solve this. 👍

Should warn or error for pkgs / files without SPDX IDs

Placeholder for now, to handle after #24 is implemented:

Should be sure that tvparser either warns or errors out upon seeing a Package or File section that is incomplete due to lack of an SPDX ID.

For example, a new package section will be started when the next PackageName tag is seen, but if it goes on to the next Package or File without seeing an SPDXID tag for that first Package, then it will not be added to the Packages map for that document. The same applies for Files (but not relevant for Snippets, see #24 (comment))

Lack of an SPDX ID makes the document invalid, but consider whether the parser should treat it as an error, a warning, and/or configurable for either of those or to optionally ignore.

rdfloader: Missing Packages from parsing example file

When running rdfloader on the canonical sample SPDX RDF file, the parsing completes without any error message. However, not all of the data appears to be getting transferred into the main data model.

The following sections are getting filled in:

• Creation Info
• Unpackaged Files
• Other Licenses
• Relationships
• Annotations

However, there are no Packages in the resulting parsed Document2_2 object. The sample document does define a couple of Packages, so these should be showing up:

https://github.com/spdx/spdx-spec/blob/6b44f663463b6f351e264435baf9a5c3b302cd83/examples/SPDXRdfExample-v2.2.spdx.rdf.xml#L258

https://github.com/spdx/spdx-spec/blob/6b44f663463b6f351e264435baf9a5c3b302cd83/examples/SPDXRdfExample-v2.2.spdx.rdf.xml#L264

https://github.com/spdx/spdx-spec/blob/6b44f663463b6f351e264435baf9a5c3b302cd83/examples/SPDXRdfExample-v2.2.spdx.rdf.xml#L1223

It also looks like it is not parsing some Snippets or File references, for example:

https://github.com/spdx/spdx-spec/blob/6b44f663463b6f351e264435baf9a5c3b302cd83/examples/SPDXRdfExample-v2.2.spdx.rdf.xml#L8

https://github.com/spdx/spdx-spec/blob/6b44f663463b6f351e264435baf9a5c3b302cd83/examples/SPDXRdfExample-v2.2.spdx.rdf.xml#L15

I note that all of these appear outside the <spdx:SpdxDocument> tag defined at line 990.

I don't know enough about RDF files to know if that's relevant, but I'm guessing it could be.

Should be able to read and write RDF files

Goal

tools-golang currently only handles SPDX files in tag-value format. It should also be capable of reading and writing SPDX files in RDF format, as an RDF format is also officially defined by the SPDX spec.

General description

Most of the functionality for tools-golang centers around the data structures defined in the spdx Golang package. Tag-value files can be read into these data structures using tvloader, and the data structures can be written to new tag-value files using tvsaver.

The objective for this enhancement is to create similar packages, rdfloader and rdfsaver, which can similarly read and write SPDX files in RDF format.

Specific details

As you'll see, the existing code in the Golang packages described above only handles version 2.1 of the SPDX specification. Let's similarly keep the focus on version 2.1 only for the RDF loading and saving, for the time being.

For new code, please mirror the suffixes "2v1" and "2_1" as they are used in the code described above. That's intended to reflect that this code is SPDX version 2.1-specific, and will (hopefully!) help make it easier to add new versions later.

It's important to me that any new code that is added should be well-tested. Please take a look at the *_test.go files in the existing directories for examples. If you are submitting new code, please be sure to include test files to go along with them.

The older version of the SPDX Golang tools, at https://github.com/spdx/ATTIC-tools-go, included RDF parsing for an older version of the SPDX specification. However, it relied on goraptor and Raptor RDF Syntax Library, the latter of which is a C library. I haven't reviewed closely, but it looks like this would require users to separately obtain and install the C library. If at all possible, I'd like for us to find a pure-Golang solution. However, that might not be possible, and I'm open to suggestions here.

tvsaver: multi-line copyright handled incorrectly

When tvsaver saves a File section with a multi-line copyright notice (FileCopyrightText), it is not wrapping it in <text></text> tags.

As a result, the generated file puts the second and subsequent lines on new lines without being wrapped in <text> tags, and the generated file is invalid and cannot be read back in.

Implement or remove tvloader/config.go

Copied over from swinslow/spdx-go#28:

In package tvloader, config.go has a Config structure that could be implemented to enable configuring the behavior of the tag-value loader. E.g., to flag it to stop on any error; to flag it to perform additional validation during the loading process; etc.

However, right now that file is unused. It should either be implemented, or else deleted.

[Optional Upgrade] Remove Duplicate Prefixes In The Field Names

Some of the structs in the tools-golang spdx data model has members that repeat the name of the struct in their name.
The member of such structs should be refactored to remove the name of the struct they are contained in.

For example,
Package struct have fields like PackageName, PackageVersion, etc.
Fields of Package struct can be renamed to Name, Version, etc.

tvsaver: missing text flags for various fields

In tvsaver, it appears that the textify() call is missing for FileNotice and FileComment:

tools-golang/tvsaver/saver2v2/save_file.go

Lines 53 to 58 in 8574b91

	if f.FileComment != "" {
	fmt.Fprintf(w, "FileComment: %s\n", f.FileComment)
	}
	if f.FileNotice != "" {
	fmt.Fprintf(w, "FileNotice: %s\n", f.FileNotice)
	}

This means that if those fields contain multi-line text, the output will be invalid because they are missing the <text></text> wrapping tags.

• Spot-check other Comments fields as well
• Fix any that are missing textify() calls

Should use go modules

Currently, tools-golang doesn't have any dependencies, so dependency management (as well as how tools-golang is imported by downstream applications) hasn't really been thought through.

tools-golang should use go modules, see [1], [2]. Specifically, it should be available for others to import as a go module.

I'm getting more familiar with the new go modules framework, but still new to it and would welcome input. It looks to me as though this should involve moving the various tools-golang packages out of the v0/ directory, as I gather that go modules don't use vX/ directories until at least a version 2 major release.

Switch to GitHub Actions

Push 0.1.0 release

Several of the currently pending PRs and issues propose significant changes to the project and API.

Before merging those, we should push a 0.1.0 release with the code in its current state, so that anyone who is currently using tools-golang won't find the API changes to be too disruptive. We are still not yet at v1.0.0 so folks should not expect the API to be stable, but I'd still like to minimize disruptions.

jsonloader: crashes if packageVerificationCodeExcludedFiles is not present

cc @specter25

In the json branch, jsonloader is crashing when trying to parse a package that contains no packageVerificationCodeExcludedFiles property in the packageVerificationCode object.

Sample file: example1-modded.txt (note that this is a JSON file, but GitHub won't let me upload it with the .json extension)

Output when running the 8-jsonloader example on this file:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x98 pc=0x4df830]

goroutine 1 [running]:
github.com/spdx/tools-golang/jsonloader2v2.JSONSpdxDocument.parseJsonPackages2_2(0xc0000a0150, 0x512216, 0x8, 0x4f00c0, 0xc0000ae120, 0xc0000ca000, 0x0, 0x0)
	/home/steve/programming/spdx/tools-golang/jsonloader2v2/parse_package.go:141 +0x2450
github.com/spdx/tools-golang/jsonloader2v2.JSONSpdxDocument.newDocument(0xc0000a0150, 0xc0000ca000, 0xda2, 0x503b80)
	/home/steve/programming/spdx/tools-golang/jsonloader2v2/jsonloader.go:88 +0x73f
github.com/spdx/tools-golang/jsonloader2v2.(*spdxDocument2_2).UnmarshalJSON(0xc0000ca000, 0xc0000c2000, 0xda1, 0xda2, 0x7f0c4dd39f70, 0xc0000ca000)
	/home/steve/programming/spdx/tools-golang/jsonloader2v2/jsonloader.go:40 +0x9d
encoding/json.(*decodeState).object(0xc0000cc000, 0x4f5a60, 0xc0000ca000, 0x16, 0xc0000cc028, 0x7b)
	/snap/go/7736/src/encoding/json/decode.go:609 +0x20a3
encoding/json.(*decodeState).value(0xc0000cc000, 0x4f5a60, 0xc0000ca000, 0x16, 0xc00009ad70, 0x4d1d4b)
	/snap/go/7736/src/encoding/json/decode.go:370 +0x6d
encoding/json.(*decodeState).unmarshal(0xc0000cc000, 0x4f5a60, 0xc0000ca000, 0xc0000cc028, 0x0)
	/snap/go/7736/src/encoding/json/decode.go:180 +0x1f4
encoding/json.Unmarshal(0xc0000c2000, 0xda1, 0xda2, 0x4f5a60, 0xc0000ca000, 0xda1, 0xda2)
	/snap/go/7736/src/encoding/json/decode.go:107 +0x11a
github.com/spdx/tools-golang/jsonloader2v2.Load2_2(0xc0000c2000, 0xda1, 0xda2, 0xda1, 0xda2, 0x0)
	/home/steve/programming/spdx/tools-golang/jsonloader2v2/jsonloader.go:22 +0xad
main.main()
	/home/steve/programming/spdx/tools-golang/examples/8-jsonloader/examplejsontotv.go:38 +0x23f

It looks like the top of the stack is at parse_package.go line 141:

tools-golang/jsonloader2v2/parse_package.go

Lines 139 to 148 in ca9e0d9

	case "packageVerificationCode":
	code := v.(map[string]interface{})
	if reflect.TypeOf(code["packageVerificationCodeExcludedFiles"]).Kind() == reflect.Slice {
	efiles := reflect.ValueOf(code["packageVerificationCodeExcludedFiles"])
	_, filename, err := extractSubs(efiles.Index(0).Interface().(string))
	if err != nil {
	return fmt.Errorf("%s", err)
	}
	pkg.PackageVerificationCodeExcludedFile = filename
	}

Note that package verification codes might not have an excluded files property, so you'll want to check whether it is present before testing or using it.

tvsaver: Missing colon for PackageVerificationCode 'excludes'

When saving out a Package that has an 'excludes' value for the PackageVerificationCode (see section 3.9 of the spec, the tag-value saver appears to incorrectly omit the ':' that the spec says should come after 'excludes'.

jsonloader: File ID should be filled into SnippetFromFileSPDXID field

cc @specter25

In the json branch, when parsing a Snippet and getting the ID of the corresponding File from the snippetFromFile property, the JSON parser is correctly associating the Snippet with the File that has that ID.

However, note that the Snippet structure also has a field where this element should be filled in. See line 14 below:

tools-golang/spdx/snippet.go

Lines 5 to 14 in ca9e0d9

	// Snippet2_1 is a Snippet section of an SPDX Document for version 2.1 of the spec.
	type Snippet2_1 struct {
	// 5.1: Snippet SPDX Identifier: "SPDXRef-[idstring]"
	// Cardinality: mandatory, one
	SnippetSPDXIdentifier ElementID
	// 5.2: Snippet from File SPDX Identifier
	// Cardinality: mandatory, one
	SnippetFromFileSPDXIdentifier DocElementID

In addition to adding the Snippet to the File.Snippets map, you'll also want to record the ID in the SnippetFromFileSPDXIdentifier field.

(There's an edge case here where the Snippet could be referring to a File that's in a different Document altogether. But that's an issue with the tag-value parser as well, so don't worry about that for now.)

Add link to pkg.go.dev documentation

https://pkg.go.dev/github.com/spdx/tools-golang

Should add to README (or somewhere) before releasing 0.3.0

Should add spdxlib utility package

Follow-on to issue #24

It would be helpful to have a package of utility functions that make it easier to work with SPDX files. In particular, now that packages etc. are stored using their SPDX identifiers as keys, it would be helpful to be able to quickly find the SPDX identifier for any DESCRIBES relationships for a document -- and/or a slice of just the single Package ID in the document, if there's just one. I'm sure there are other useful utility functions like this that would be helpful.

[Optional Upgrade] Checksum Struct

Checksums are currently stored in coalesced strings. They can be upgraded to use a struct.

Following seems like a suitable implementation:

type Checksum {
    Algorithm, Value String
}

For now, the Algorithm field is kept as a String.

We can also make it a field that allows only algorithms defined by the model in an enum.
Implementation goes as follows:

type ChecksumAlgorithm int
const (
    MD5 ChecksumAlgorithm = iota
    SHA1
    SHA512
    MD2
    // other algorithms...
)

type Checksum {
    Algorithm ChecksumAlgorithm
    Value String
}

Prepare 0.2.0 release

Key items before release:

• Review and test json branch, closing fixed bugs
• Merge in json branch with support for reading json files
• Check other open issues
• Close issue re: API / field name changes
• Update README
• Add release notes

Should handle SPDX v2.2

Version 2.2 of the SPDX spec is due to be published shortly (current draft is available at: https://spdx.github.io/spdx-spec/v2-draft/)

All parts of spdx/tools-golang should be able to handle 2.2 documents.

Additionally, ideally spdx/tools-golang should include functionality to translate an in-memory SPDX 2.1 document to 2.2, and vice versa (in the latter case, losing information where necessary for fields, relationships, etc. that don't exist in 2.1).

rdfloader: Issue With The Licenses

Licenses, when associated with itself, will cause infinite recursion in the tool.

Licenses are implemented without using pointers. Hence, everytime a new License object is created.
It is not possible to cache it. As, everytime a new object will be created and adding data to it won't affect the old one causing inconsistency of data.

@swinslow , Is it possible for any spdx document to have cyclic dependency on licenses?
Only way I can see that happening is via a ConjunctiveLicenseSet or DisjunctiveLicenseSet.

@swinslow, should I assume that there might be a case where a license is associated with itself in some way?

Consider changing data model to IDs as keys

Currently, the data model for spdx/tools-golang stores everything in slices. For example:

• Document2_1 contains slices of packages, other license sections, relationships, etc.
• Package2_1 contains a slice of Files

Although this makes it easy to maintain the ordering of the applicable elements, it also makes lookup extremely difficult. Finding an element with a particular SPDX ID requires walking through the entire slice.

A better method might be for the internal data representation to store SPDX elements in maps, where the SPDXID (or LicenseRef ID) is the key.

If ordering is important to maintain, then an "order" field could also be maintained when reading.

tvloader: Cannot read Relationships after Other Licenses

Currently, tvloader will fail if it sees a Relationship tag while parsing an OtherLicense section. It apparently assumes that any Relationships will appear prior to moving into parsing Other Licenses.

As a result, it cannot parse files with Relationship tags following OtherLicense sections -- even though this is valid syntax (and is even what gets generated by tvsaver).

jsonloader: typo in filenames

cc @specter25

In the json branch, there are files at:

• jsonloader2v2/parse_reationship.go
• jsonloader2v2/parse_reationship_test.go

relationship is misspelled as reationship in these filenames. They should be corrected (a simple git mv should take care of it).

jsonloader: Snippet byte ranges are parsed into the wrong field

cc @specter25

In the json branch, when parsing Snippets, it is incorrectly assigning the Byte start/end data into the Line start/end variables. See lines 68 and 69 below:

tools-golang/jsonloader2v2/parse_snippets.go

Lines 57 to 70 in ca9e0d9

	if reflect.TypeOf(v).Kind() == reflect.Slice {
	info := reflect.ValueOf(v)
	lineRanges := info.Index(0).Interface().(map[string]interface{})
	lineRangeStart := lineRanges["endPointer"].(map[string]interface{})
	lineRangeEnd := lineRanges["startPointer"].(map[string]interface{})
	snippet.SnippetLineRangeStart = int(lineRangeStart["lineNumber"].(float64))
	snippet.SnippetLineRangeEnd = int(lineRangeEnd["lineNumber"].(float64))
	byteRanges := info.Index(1).Interface().(map[string]interface{})
	byteRangeStart := byteRanges["endPointer"].(map[string]interface{})
	byteRangeEnd := byteRanges["startPointer"].(map[string]interface{})
	snippet.SnippetLineRangeStart = int(byteRangeStart["offset"].(float64))
	snippet.SnippetLineRangeEnd = int(byteRangeEnd["offset"].(float64))
	}

tvsaver: check multiline <text> tag wrapping

From some recent testing on saving tag-value documents, it looks like some multiline fields are not getting correctly wrapped with <text>...</text> tags when they run over more than a single line.

In particular, I've noticed it for the following fields, when reading from an RDF file (the canonical RDF sample file) and saving out as a 2.2 tag-value file:

• FileComment
• FileNotice
• LicenseComment

Inconsistent Expectation and Implementation

According to spdx/tools-golang/spdx/identifier.go#L8, the ElementID must not contain "SPDX-Ref-" portion.
But while setting SPDXID in tvloader, parse_creation_info.go#L29 sets it as it is without trimming the 'SPDX-Ref'- portion. Also, the test here, asserts if the SPDXIdentifier is "SPDXRef-DOCUMENT" and not "DOCUMENT"

jsonsaver: omit empty lists rather than null

As flagged in #100 (thank you @yanyag!), currently jsonsaver will output null for properties such as files, snippets and externalDocumentRefs if there are none.

Similarly, if there is no excludes item for packageVerificationCodeExcludedFiles, it will output [""].

Looking at the examples from the spdx-spec and spec-examples repos, it looks like these JSON properties should typically be omitted if they are empty, rather than saved with null.

examples: Add a comment in the `examples/` folder for each example program explaining how to run it

Add a comment in the examples/ folder for each example program explaining how to run it.

It is desirable that these comments be added after Pull Request #114 is successfully accepted because the first comment of this kind was added in this PR.

Running examples are required for the following examples:

• examples/1-load
• examples/2-load-save
• examples/4-search
• examples/5-report
• examples/6-licensediff
• examples/7-rdfloader
• examples/8-jsontotv
• examples/9-tvtojson
• examples/10-jsonloader

It would be good if these examples were also put in the file examples/ README.md

Handle additional checksums permitted by 2.2

In SPDX 2.1 and earlier, the only hash algorithms that were permitted for e.g. File checksums, Package checksums, etc. were SHA1 (mandatory), SHA256 (optional) and MD5 (optional). Because of this, those fields were represented as three separate fields, one for each checksum type (see e.g. here).

In SPDX 2.2, several additional algorithms were permitted to be used (see section 4.4 of the spec). The tools-golang model doesn't currently allow these to be used.

Rather than adding more optional fields for each algorithm, I'm more inclined to switch to something where there is a Checksum type defined, with something like the following:

type Checksum2_2 struct {
    Type ChecksumType
    Value string
}

and then using a series of consts / enums to represent the different algorithms:

// should this all be ChecksumType2_2, etc. instead of just ChecksumType?
type ChecksumType int

const (
    CHECKSUM_TYPE_SHA1 = iota
    CHECKSUM_TYPE_SHA256
    // . . .
)

and then changing the various Checksum fields in Package, File, etc. to be a slice of Checksum2_2.

@RishabhBhatnagar I'd be interested in your thoughts on this approach!

Update code and docs for move to spdx/tools-golang

The code and documentation still make reference to the project's original location at https://github.com/swinslow/spdx-go. These should all be updated to refer to the new spdx/tools-golang repo.

This likely includes code within some of the packages to, e.g., include the tool's name in new SPDX documents.

Basically, anything where the strings swinslow or spdx-go appear should likely be changed. =)

Get OpenSSF Best Practices badge

https://bestpractices.coreinfrastructure.org/en

Long-term project; look into what is required in terms of technical as well as community enhancements to qualify for the badge.

jsonloader: "excludes" in packageVerificationCode

cc @specter25

In the json branch, when parsing the packageVerificationCode object, it appears that the current JSON parser is expecting an excludes: tag to be present within the array elements:

tools-golang/examples/8-jsonpasrser2v2/test.json

Lines 99 to 102 in 8a86c1f

	"packageVerificationCode" : {
	"packageVerificationCodeExcludedFiles" : [ "excludes: ./package.spdx" ],
	"packageVerificationCodeValue" : "d6a770ba38583ed4bb4525bd96e50461655d2758"
	},

For parsing tag-value documents, that would be correct: the tag-value spec requires excludes: to be present.

However, looking at the sample JSON file from the SPDX spec, excludes: is not present and the array element is just the filename itself:

https://github.com/spdx/spdx-spec/blob/3dddfc34d7e770d4c17fc19f61be8554e3934c3d/examples/SPDXJSONExample-v2.2.spdx.json#L100-L103

Also, when running some tests using the SPDX Online Tools converter, it appears to me that the version from the spec samples is also used there; excludes: is not present.

So, I believe that when parsing that field, you won't want to look for an excludes: tag in the array element, and instead you'll want to just take the array element directly as the filename.

Tvsaver : invalid tag SnippetSPDXIdentifier

cc @swinslow

The tag-value saver has a parses Snippet SPDX Id's incorrectly using the tag SnippetSPDXIdentifier whereas it should be SnippetSPDXID as defined in the SPDX specification .

Prepare 0.3.0 release

Key items before release:

• Review and test json branch
• Merge in json branch with support for saving json files
• Address issue #103
• Address issue #107
• Address issue #116
• Review and edit new docs
• Check other open issues
• Update README
• Publish -rc1
• Add release notes
• 1 week for feedback on -rc1, then:
  • update release notes for recent changes (also note branch protection added)
  • push v0.3.0

Builder uses local time rather than UTC for CreationInfo.Created

Copied over from swinslow/spdx-go#7:

Currently, the builder package uses time.Now() to get the time for CreationInfo.Created. According to 2.9.1, it should be in UTC. Note that just calling UTC() as part of getting the time doesn't solve this, presumably because it needs more info about the user's local time zone.

Should be able to read and write JSON files

SPDX 2.2 will include a new JSON format: see https://github.com/spdx/spdx-spec/blob/development/v2.2/examples/SPDXJSONExample-v2.2.spdx.json

tools-golang should support reading from and writing to SPDX documents in JSON format. This will probably require a translation between the tools-golang data model and an intermediate model for the JSON format.

[Optional Upgrade] SpdxItem and SpdxElement

As evident from the UML diagram, classes like Package, File, Snippets, etc derive some of their fields/properties from an abstract class called SpdxItem.

Currently, each of the subclasses defines each of those properties in their own structs. I suggest pulling out the common fields and create a new struct type "SpdxItem" which will hold those common attributes. The subclasses can then use composition to include those fields in their own definition.
This will allow the model to have less entry points for common fields allowing

Similar is the argument for SpdxElement

Sample Working:

type SpdxItem struct {
    licenseConcluded AnyLicenseInfo
    licenseInfoFromFiles SimpleLicenseInfo
    copyrightText String
    licenseComments String
}

type Package struct {
    SpdxItem
    summary String
    description String
    // ...other fields...
}

type File struct {
    SpdxItem
    fileType FileType
    checksum Checksum
    // ...other fields...
}

tvloader: not parsing Package after unpackaged file

When using tvloader to parse https://github.com/spdx/spdx-spec/blob/development/v2.2/examples/SPDXTagExample-v2.0.spdx (with a change to make the spdx version 2.1), it should detect the glibc Package with ID SPDXRef-Package.

However, the resulting document does not appear to include this Package, and all files in the document are being considered as unpackaged.

Ambiguity in ExternalDocumentRef of the spdx data model

tools-golang/spdx/creation_info.go#L31 expects the ExternalDocumentRefs to be a list of strings.
But from the spdx-spec of externalDocumentRef for RDF file, it can be seen that externalDocumentRef is made up of two properties viz:

1. SPDXDocument &
2. checksum

Spdx data model must be modified to accept ExternalDocumentReferences as a List of struct having two described attributes and not a List of strings.