microsoft / ossgadget Goto Github PK

Collection of tools for analyzing open source packages.

License: MIT License

C# 97.33% Dockerfile 0.25% Shell 0.50% HTML 1.91%

ossgadget's Introduction

OSS Gadget

Note: OSS Gadget is currently in public preview and is not ready for production use.

OSS Gadget is a collection of tools that can help analyze open source projects. These are intended to make it simple to perform low-level tasks, like locating the source code of a given package, downloading it, performing basic analyses on it, or estimating its health. The tools included in OSS Gadget will grow over time.

Included Tools

A list of tools included is below. Click on the name of a tool to go to the wiki for usage information.

oss-characteristic: Identify a package's notable characteristics and features. Uses Application Inspector.
oss-defog: Searches a package for obfuscated strings (Base-64).
oss-detect-backdoor: Identifies potential backdoors and malicious code within a package. Currently has a high false-positive rate.
oss-detect-cryptography: Identifies cryptographic implementations within a package.
oss-diff: Compares two packages using a standard diff/patch view.
oss-download: Downloads a package and extracts it locally.
oss-find-domain-squats: Identifies potential typo-squatting for a given domain name.
oss-find-source: Attempts to locate the source code (on GitHub, currently) of a given package.
oss-find-squats: Identifies potential typo-squatting for a given package.
oss-health: Calculates health metrics for a given package.
oss-metadata: Retrieves metadata from deps.dev or libraries.io for a given package.
oss-risk-calculator: Calculates a metric for risk of using a package.
oss-reproducible: Evaluates a package for semantic equivalency with the detected source code to produce an estimate of how well the two comport.

All OSS Gadget tools accept one or more Package URLs as a way to uniquely identify a package. Package URLs look like pkg:npm/express or pkg:gem/[email protected]. If you leave the version number off, it implicitly means, "attempt to find the latest version". Using an asterisk (pkg:npm/express@*) means "perform the action on all available versions".

Package Sources

OSS Gadget supports packages provided by these sources:

Cargo - pkg:cargo/...
Cocoapods - pkg:cocoapods/...
Composer - pkg:composer/...
CPAN - pkg:cpan/...
CRAN - pkg:cran/...
GitHub - pkg:github/...
Go - pkg:golang/...
Hackage - pkg:hackage/...
Maven - pkg:maven/...
NPM - pkg:npm/...
NuGet - pkg:nuget/...
RubyGems - pkg:gem/...
PyPI - pkg:pypi/...
Ubuntu - pkg:ubuntu/...
Visual Studio Marketplace - pkg:vsm/...
Generic - pkg:url/...?url=URL

We will continue expanding this list to cover additional package management systems and would be happy to accept contributions from the community.

Basic Usage

All OSS Gadget tools are command line programs. When installed globally, they can be accessed from your path. For example, to download the NPM left-pad module, type:

$ oss-download pkg:npm/left-pad

This will download left-pad into a newly-created directory named [email protected]. (Because, at the time of this writing, 1.3.0 was the latest version of left-pad).

Each of the programs self-documents information on command line options (--help).

Building from Source

OSS Gadget builds with standard dotnet build commands and includes tests via dotnet test.

See Building from Source in the wiki for information on building from source.

Docker Image

See Docker Image in the wiki for information on how to use the included Dockerfile.

Advanced Usage

See Advanced Usage in the wiki for advanced usage information like changing API endpoints.

Reporting Security Vulnerabilities

To report a security vulnerability, please see SECURITY.md.

Contributing to OSS Gadget

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

ossgadget's People

Contributors

Stargazers

Watchers

ossgadget's Issues

Support Statement Clarification

We are piloting a new support statement, to make it clear to repo users where/how to ask questions about the usage of the software in this repo. Our goal is to eventually have a statement like this in most repos. Do you have any feedback about the statement? Please share it here.

Crypto ML on Images of Code

See: https://gist.github.com/naveedmurtuza/6600103 for converting text to image.

Then use machine learning on the images to classify code that contains crypto or not.

PackageURL implementation does not recognize packages of the form pkg:npm/@types/react

The parsing logic need to recognize the above name format, which is supposed to be valid:
pkg:npm/@types/react => https://www.npmjs.com/package/@types/react
It gives an error as of now:
: Invalid purl: Does not contain a minimum of a 'type' and a 'name'

Project health clarification

I've noticed that some projects have 0.0% for issues or security.

Is this due to a lack of historical data in which to determine health? If so, then what would be an alternative way to display this info as the lack of evidence seems to imply a project is doing poorly which is not the case. And likewise, a lack of evidence should not necessarily be justification to make the project look good either.

Thoughts?

MultiExtractor: Extract Disk Images

https://github.com/discutils/discutils - this support .ISO, .DMG, .VHD etc.

This library also implements .ar parsing which can probably replace our custom GnuAr parser.

Improve version number sorting (use publish date if possible)

Example: pkg:composer/ircmaxell/random-lib

Result: Gets dev-improved_string_presets.
Should: Get v1.2.0

Unable to update DevSkim to OAT-enabled version

When we update the DevSkim dependency to the latest version, we get a few errors:

https://github.com/microsoft/OSSGadget/blob/main/src/oss-detect-cryptography/DetectCryptographyTool.cs#L408

rules.Count() fails because _rules is not populated, only _oatRules. I think Count() should reflect _oatRules instead.

https://github.com/microsoft/OSSGadget/blob/main/src/oss-detect-cryptography/DetectCryptographyTool.cs#L478

The call into Analyze actually completes, but doesn't give the expected results. I was having a hard time tracing to see where we were getting different results.
Repro:
Using DevSkim v0.4.144

PS C:\Users\scovetta\source\repos\OSSGadget\src\oss-detect-cryptography\bin\Debug\netcoreapp3.1> .\oss-detect-cryptography.exe pkg:npm/md5.js
Microsoft OSS Gadget - oss-detect-cryptography 0.1.0.0
[X] pkg:npm/md5.js - This software package appears to implement Hash.MD5.
[ ] pkg:npm/md5.js - This software package does NOT have a high-density of cryptographic operators.
[ ] pkg:npm/md5.js - This software package does NOT contains words that suggest cryptography.

Using DevSkim v0.4.188

PS C:\Users\scovetta\source\repos\OSSGadget\src\oss-detect-cryptography\bin\Debug\netcoreapp3.1> .\oss-detect-cryptography.exe pkg:npm/md5.js
Microsoft OSS Gadget - oss-detect-cryptography 0.1.0.0
[ ] pkg:npm/md5.js - This software package does NOT appear to implement cryptography.

Crash in oss-detect-backdoor.exe with no arguments passed

Getting an unhandled exception, expecting to get the help menu if no args passed in.

PS C:\OSSGadget\src\oss-detect-backdoor\bin\Debug\netcoreapp3.1> .\oss-detect-backdoor.exe
Unhandled exception. System.InvalidCastException: Unable to cast object of type 'System.Runtime.CompilerServices.NullableAttribute' to type 'CommandLine.Text.UsageAttribute'.
   at CommandLine.Core.ReflectionExtensions.<>c.<GetUsageData>b__2_3(<>f__AnonymousType1`2 <>h__TransparentIdentifier0)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source)
   at CommandLine.Core.ReflectionExtensions.GetUsageData(Type type)
   at CommandLine.Text.HelpText.GetUsageFromType(Type type)
   at CommandLine.Text.HelpText.RenderUsageTextAsLines[T](ParserResult`1 parserResult, Func`2 mapperFunc)+MoveNext()
   at CSharpx.EnumerableExtensions.ToMaybe[T](IEnumerable`1 source)
   at CommandLine.Text.HelpText.AutoBuild[T](ParserResult`1 parserResult, Func`2 onError, Func`2 onExample, Boolean verbsIndex, Int32 maxDisplayWidth)
   at CommandLine.Text.HelpText.AutoBuild[T](ParserResult`1 parserResult, Func`2 onError, Int32 maxDisplayWidth)
   at Microsoft.CST.OpenSource.OSSGadget.DisplayHelp[T](ParserResult`1 result, IEnumerable`1 errs) in C:\Users\scovetta\source\repos\OSSGadget-GitHub\src\Shared\OSSGadget.cs:line 33
   at Microsoft.CST.OpenSource.OSSGadget.<>c__DisplayClass7_0`1.<ParseOptions>b__0(IEnumerable`1 errs) in C:\Users\scovetta\source\repos\OSSGadget-GitHub\src\Shared\OSSGadget.cs:line 55
   at CommandLine.ParserResultExtensions.WithNotParsed[T](ParserResult`1 result, Action`1 action)
   at Microsoft.CST.OpenSource.OSSGadget.ParseOptions[T](String[] args) in C:\Users\scovetta\source\repos\OSSGadget-GitHub\src\Shared\OSSGadget.cs:line 55
   at Microsoft.CST.OpenSource.DetectBackdoorTool.Main(String[] args) in C:\Users\scovetta\source\repos\OSSGadget-GitHub\src\oss-detect-backdoor\DetectBackdoorTool.cs:line 56
   at Microsoft.CST.OpenSource.DetectBackdoorTool.<Main>(String[] args)

Add a "Save" option to RecursiveExtractor

So you could for example open up an archive modify the contents and then write it back again. Would be cool if this still preserves recursive/original files structures/formats implicitly discovered from the files.

Needs adding a File Extension check to each FileEntry and see if its an archive that needs to be created etc.

Add more disc formatting support to VHD/VHDX reading

DiscUtils looks like it supports:

https://github.com/DiscUtils/DiscUtils/blob/b3ffb5123eea8791a90d6a2beb65c68cc5708b24/Library/DiscUtils.FileSystems/DiscUtils.FileSystems.csproj#L11-L18
Btrfs
HfsPlus (worth doing alongside #63)
SquashFs
Xfs

in addition to the EXT, FAT and NTFS support we already have.

Add ranking to oss-find-source output

When there are multiple GitHub repositories listed, we should try to rank them by how likely they are the "right" source code.

Support Parallel RAR Extraction

Already implemented but currently disabled because it behaves unreliably in the pipeline.

oss-detect-backdoor fails with error message on all packages

Linux:

./oss-detect-backdoor pkg:npm/express
Error analyzing /tmp/dgzxagdo.gob/[email protected]: Invalid rule path Resources\BackdoorRules

Windows:

.\oss-detect-backdoor.exe pkg:npm/express
�[35mError analyzing C:\Users\sujacob\AppData\Local\Temp\5njmzwnf.abr\[email protected]: Duplicate ruleId BD000200 found.�[0m

Add JSON output to all tools

Each tool should be able to output structured (JSON) status information.

Create an animated screenshot of using OSS Gadget

We should have a short screenshot (animated .gif) showing how to use OSS Gadget.

Enable support for Foo::Bar packages on CPAN

Most Perl packages are known as Foo::Bar rather than Foo-Bar. It would be nice to specify it that way unless it breaks PackageURL.

Expanded .ar format support

https://github.com/xoofx/LibObjectFile

Supports BSD, GNU and DEB formats. Doesn't appear to have a way to distinguish directly from the stream, looks like you need to know which kind it is. Will need to implement a checker either in LibObjectFile library (probably better) or in MiniMagic.

We have an .ar of each type checked into the TestData folder:
Common: Like Deb, already available in test data as Shared.deb
GNU: Available in test data as Shared.ar
BSD: Available in test data as Shared.bsd.ar

Builds Failing - NU1101

Our build pipeline started failing with:

src\oss-defog\oss-defog.csproj(0,0): Error NU1101: Unable to find package Nerdbank.GitVersioning. No packages exist with this id in source(s): Microsoft Visual Studio Offline Packages

Output: https://github.com/microsoft/OSSGadget/pull/37/checks?check_run_id=597385821

Publish a Docker Image with all of our CST tools

Putting this here as it's our most cross project repo.

In addition to the project specific docker containers we are planning to create it seems like we could publish a meta package with all of our tools for someone who may want to use multiple tools in one go.

I'm thinking this could include all our .net tools
AttackSurfaceAnalyzer
DevSkim
ApplicationInspector
OSSgadget tools
Multiextractor

.vhdx Extraction failing

.vhdx extraction tests are failing and have been disabled.

First failed pipeline run I see is from commit 3e781fa75d948e438668c08be834ed3b98f2f780.

ExtractArchive (Shared.vhdx,True)

Test method Microsoft.CST.OpenSource.Tests.ExtractorTests.ExtractArchive threw exception:

System.NullReferenceException: Object reference not set to an instance of an object.
at DiscUtils.Streams.WrappingMappedStream`1.get_Length()
at DiscUtils.Streams.SubStream..ctor(Stream parent, Int64 first, Int64 length)
at DiscUtils.Partitions.GuidPartitionTable.Open(GptEntry entry)
at DiscUtils.Partitions.GuidPartitionInfo.Open()
at DiscUtils.PhysicalVolumeInfo.Open()
at DiscUtils.LogicalVolumeInfo.Open()
at DiscUtils.FileSystemManager.DetectFileSystems(VolumeInfo volume)
at Microsoft.CST.OpenSource.MultiExtractor.Extractor.DumpLogicalVolume(LogicalVolumeInfo volume, String parentPath, Boolean parallel, FileEntry parent)+MoveNext() in d:\a\1\s\src\MultiExtractor\Extractor.cs:line 1269
at Microsoft.CST.OpenSource.MultiExtractor.Extractor.ExtractVHDXFile(FileEntry fileEntry, Boolean parallel)+MoveNext() in d:\a\1\s\src\MultiExtractor\Extractor.cs:line 478
at Microsoft.CST.OpenSource.MultiExtractor.Extractor.ExtractFile(String filename, Boolean parallel)+MoveNext() in d:\a\1\s\src\MultiExtractor\Extractor.cs:line 211
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Microsoft.CST.OpenSource.Tests.ExtractorTests.ExtractArchive(String fileName, Boolean parallel, Int32 expectedNumFiles) in d:\a\1\s\src\tests\ExtractorTests.cs:line 58

ExtractArchive (Shared.vhdx,False)

Test method Microsoft.CST.OpenSource.Tests.ExtractorTests.ExtractArchive threw exception:

Hybrid Parallel-Enumerable Extractor

Try to gain some of the benefits of parallelization without having to load the entire extracted contents of the archive into memory before returning.

Create an internal queue to push to.

This continually fills up the stack until the file is fully extracted, but memory is freed as the caller pulls off the enumerable.
Allows callers to start pulling entries immedietly instead of waiting for all processing to be done.
Frees up memory as objects are yielded.
This subverts the expectations of what an enumerable does. It will continue processing between yields.

ZipFile zipFile = new ZipFile(fileEntry.Content);
var zipEntries = new List<ZipEntry>();
foreach (ZipEntry zipEntry in zipFile)
{
    zipEntries.Add(zipEntry);
}
ConcurrentStack<FileEntry> files;
bool done = false;
_ = Task.Run(() => {
    Parallel.ForEach(zipEntries, zipEntry => 
    {
        using var memoryStream = new MemoryStream();
        byte[] buffer = new byte[BUFFER_SIZE];
        var zipStream = zipFile.GetInputStream(zipEntry);
        StreamUtils.Copy(zipStream, memoryStream, buffer);    
        var newFileEntry = new FileEntry(zipEntry.Name, fileEntry.FullPath, memoryStream);
        files.PushRange(ExtractFile(newFileEntry, true))
    });
    done = true;
});
while (!done){
    if (files.TryPop(out FileEntry result)){
        yield return result;
    }
    else{
        Thread.Sleep(1);
    }
}

Process by chunks

Do parallel processing on a chunk of files at a time, stopping in between chunks to yield until empty.

ZipFile zipFile = new ZipFile(fileEntry.Content);
var zipEntries = new List<ZipEntry>();
foreach (ZipEntry zipEntry in zipFile)
{
    zipEntries.Add(zipEntry);
}
ConcurrentStack<FileEntry> files;
while (zipEntries.Count > 0)
{
    if (zipEntries.Count > MIN_BATCH_SIZE){
        int batchSize = Math.min(MAX_BATCH_SIZE,zipEntries.Count);
        Parallel.ForEach(zipEntries[0..batchSize], zipEntry => 
        {
            using var memoryStream = new MemoryStream();
            byte[] buffer = new byte[BUFFER_SIZE];
            var zipStream = zipFile.GetInputStream(zipEntry);
            StreamUtils.Copy(zipStream, memoryStream, buffer);    
            var newFileEntry = new FileEntry(zipEntry.Name, fileEntry.FullPath, memoryStream);
            files.PushRange(ExtractFile(newFileEntry, true))
        });
        zipEntries.RemoveRange(0, batchSize);
    }
    else{
        foreach (var zipEntry in zipEntries){
            using var memoryStream = new MemoryStream();
            byte[] buffer = new byte[BUFFER_SIZE];
            var zipStream = zipFile.GetInputStream(zipEntry);
            StreamUtils.Copy(zipStream, memoryStream, buffer);    
            var newFileEntry = new FileEntry(zipEntry.Name, fileEntry.FullPath, memoryStream);
            foreach (var extractedFile in ExtractFile(newFileEntry, parallel))
            {
                yield return extractedFile;
            }        
        }
        zipEntries.Clear();
    }
    while (files.TryPop(out FileEntry entry))
    {
        yield return entry;
    }
}

multiextractor requires consumers to modify their nuget.config

This should not be required perhaps we can include the dlls directly in the multi extractor nupkg.

Standardize Code Formatting

For code formatting.

https://github.com/dotnet/format

== default vs. == null

I thought I read somewhere that doing:

if (thing == default) ...  <-- Good
if (thing == null) ... <-- Not as Good

I think I was mistaken here. Leaving this issue open to track cleanup across the codebase.

Catch Nested Quines

Problem

Currently we check if an archive contains itself (quine) but we don't capture a nested quine like:

A contains B which contains A.

I don't have a sample of a quine bomb like that but I believe it is theoretically possible. I believe our current byte based system will still catch this, but likely not very quickly.

Proposed Solution

Pass a list of FileEntries of parents (optionally) when calling ExtractFile, check if the newly extracted file is:

Named identically
The same size
bytewise equals to any of its parents

If so, this is a quine.

add a multiextractor cli

To be published as a dotnet tool with #85

multiextractor --src archive.anytype --dest path/to/create

Also

multiextractor archive.anytype is the same as multiextractor --src archive.anytype --dest archive.anytype.extraction/

Add SDL/1CS Task to Pipeline

Need the following tasks to be added to the core or a new sdl pipeline:
credscanner
codeinspector

i.e.

task: CredScan@3
inputs:
outputFormat: 'sarif'
task: CodeInspector@2
inputs:
ProductId: 'e6121b8f-ffd8-40b1-981d-a5ea5c121baa'

oss-download deletes directory after downloading component

When oss-download downloads a package, it should land by default in a new directory:

PS C:\ossgadget> .\oss-download.exe pkg:pypi/requests
Microsoft OSS Gadget - oss-download 0.1.0.0
Downloaded pkg:pypi/requests to C:\ossgadget\[email protected]
Downloaded pkg:pypi/requests to C:\ossgadget\[email protected]

But the directory isn't there, because it's deleted:

2020/07/10 22:58:22.290|TRACE|Removing directory C:\ossgadget\[email protected] |
2020/07/10 22:58:22.312|TRACE|Removing directory C:\ossgadget\[email protected] |

Add unit tests

We need to add unit tests, and make sure they run automatically as part of CI.

tests failing after pipeline changes

Hi,

Could you please check why the tests started failing after the checkins here:
https://twcsecurityassurance.visualstudio.com/SecurityEngineering/_build/results?buildId=6812&view=results

There seems to be no changes to the test code or application logic in this checkin.
https://twcsecurityassurance.visualstudio.com/SecurityEngineering/_build?definitionId=117&_a=summary

Please make sure the tests are succeeding for every PR/checkin...also if it is not related to this checkin, please assign it back to me.

Thanks,

oss-characterstics: Extra key in languages

>.\oss-characteristic.exe pkg:npm/express
pkg:npm/express
Programming Language: package.json, javascript
Unique Tags:
 * [Dependency.SourceInclude, 0]
 * [OS.Network.Connection.Http.Ajax, 0]
 * [Metadata.Application.Author, 0]
 * [Data.Sensitive.Secret, 0]
 * [Metadata.Application.Description, 0]
 * [Data.Media.Audio, 0]
 * [Metadata.CloudServices.Code.Repo.GitHub, 0]
 * [Metadata.Application.Version, 0]
 * [OS.Network.Connection.Http, 0]
 * [OS.Network.Connection.General, 0]
 * [Metadata.Application.Type.Web.Service, 0]
 * [Data.Sensitive.Credentials, 0]
 * [Cryptography.Encryption.General, 0]
 * [Data.Sensitive.Identification, 0]

The Programming language displays invalid entry: packages.json

Release MultiExtractor nupkg

Create DMG example

Can someone provide or point me to an example for how to create a DMG image?

Implement deepsearch for finding the source code repository of a package

Search open source repositories, and compare the names (and optionally contents) of the package to figure out the source code.

oss-download creates an extra directory, mis-named

Example:

oss-download pkg:npm/left-pad

You get:
/npm-left-pad/1.3.0/[email protected]/package

That middle [email protected] is wrong -- either it shouldn't be there at all, or it should at least have the correct version number.

Mock Downloader for tests

The download tests can take a long time to run (just took me 11 minutes). The main culprit is VSM_Download_Version_Succeeds which took half of the time at 5.5 minutes.

Maven_Download_Version_Succeeds also took 4 minutes.

I think we don't need to test actual network traffic and we should try to mock the actual downloading if we can.

This also applies to the detectcryptography tests which could be run against cached copies of the code as well.

Release Notarized Mac OS Binaries

There are a few known issues with MacOS:

On the latest version of MacOS, notarization will be a problem unless the user turns it off. We need to get our builds notarized.
On older versions of MacOS (e.g. 10.10), the binaries do not run. This is very likely because .NET Core 3+ only supports MacOS 10.13+ as per this link.
The executables aren't chmod'ed correctly. Doing chmod +x oss-download, for example, fixes that temporarily.

Download tests are failing

X VSM_Download_Version_Succeeds [4s 843ms]
  X VSM_Download_Version_Succeeds (pkg:vsm/ms-vscode/PowerShell,extension.vsixmanifest,8) [4s 843ms]
  Error Message:
   Assert.AreEqual failed. Expected:<8>. Actual:<1>. 
  Stack Trace:
     at Microsoft.CST.OpenSource.Tests.DownloadTests.TestDownload(String purl, String targetFilename, Int32 expectedCount) in D:\a\1\s\src\tests\DownloadTests.cs:line 168
   at Microsoft.CST.OpenSource.Tests.DownloadTests.VSM_Download_Version_Succeeds(String purl, String targetFilename, Int32 expectedCount) in D:\a\1\s\src\tests\DownloadTests.cs:line 115
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.ThreadOperations.ExecuteWithAbortSafety(Action action)

  X Invalid_Package_Test_Download [30s 27ms]
  X Invalid_Package_Test_Download (pkg:blah/blah,,1) [30s 27ms]
  Error Message:
   Assert.ThrowsException failed. Threw exception ArgumentNullException, but exception ArgumentException was expected. Expected a ArgumentException but no exception was thrown.
Exception Message: Value cannot be null. (Parameter 'searchPattern')
Stack Trace:    at System.IO.Directory.InternalEnumeratePaths(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options)
   at System.IO.Directory.EnumerateFiles(String path, String searchPattern, SearchOption searchOption)
   at Microsoft.CST.OpenSource.Tests.DownloadTests.TestDownload(String purl, String targetFilename, Int32 expectedCount) in D:\a\1\s\src\tests\DownloadTests.cs:line 165
   at Microsoft.CST.OpenSource.Tests.DownloadTests.<>c__DisplayClass15_0.<<Invalid_Package_Test_Download>b__0>d.MoveNext() in D:\a\1\s\src\tests\DownloadTests.cs:line 135
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.ThrowsExceptionAsync[T](Func`1 action, String message, Object[] parameters)
  Stack Trace:
     at Microsoft.CST.OpenSource.Tests.DownloadTests.Invalid_Package_Test_Download(String purl, String targetFilename, Int32 expectedCount) in D:\a\1\s\src\tests\DownloadTests.cs:line 133
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.ThreadOperations.ExecuteWithAbortSafety(Action action)

See: https://twcsecurityassurance.visualstudio.com/SecurityEngineering/_build/results?buildId=6324&view=logs&jobId=09505e15-86fb-54ab-7e0b-0aff55c56acc&j=09505e15-86fb-54ab-7e0b-0aff55c56acc&t=4b4f01c2-f30b-50d9-b044-d0867ad54f9f

Better console feedback on errors

For oss-health, if GH token isn't defined, it fails silently, though errors still go to oss-gadget.log.

Download tool not recognizing already downloaded (cached) versions

E:\test>...\oss-defog\bin\Debug\netcoreapp3.1\oss-defog.exe pkg:npm/[email protected]

E:\test>dir
...

 Directory of E:\test

05/08/2020  08:34 PM    <DIR>          .
05/08/2020  08:34 PM    <DIR>          ..
05/08/2020  08:34 PM    <DIR>          [email protected]

E:\test>...\oss-defog\bin\Debug\netcoreapp3.1\oss-defog.exe pkg:npm/[email protected]

E:\test>dir

 Directory of E:\test

05/08/2020  08:34 PM    <DIR>          .
05/08/2020  08:34 PM    <DIR>          ..
05/08/2020  08:34 PM    <DIR>          [email protected]
05/08/2020  08:34 PM    <DIR>          [email protected]

Only publish new versions of MultiExtractor to NuGet if MultiExtractor has changed.

Problem

Currently, OSS Gadget Characteristic depends on Application Inspector which depends on MultiExtractor.

Here's what happens when we update anything in OSS Gadget:

On every update to any OSS Gadget component we publish a new version of all the OSS Gadget tools including Multiextractor.
To be up to date, we then need to update the reference in Application Inspector for MultiExtractor. This publishes a new version of Application Inspector.
We then need to update OSS Gadget Characteristics with the reference to the new version of Application Inspector. Go to 1.

Proposed Solution

Only publish a new version of each tool to NuGet if it in particular has changed. In the above example, instead of publishing a new version of MultiExtractor, only Characteristic tool has changed and we skip the dependency loop.

Possible implementation

This could be done with separate sections for each OSS Gadget tool in the pipeline.

It may still be desirable to publish the full set to GitHub, but that adds some complexity. It may be fine to only publish the tools that have changed.

Additional Benefits

Less version churn (especially no-op version upgrades) for users who are only referencing one library or tool - for example just MultiExtractor.

Complete publishing pipeline for MacOS, Linux, .NET Core

We had to modify the pipeline to get around some errors that we were having with dotnet publish. The workaround involved adding a <runtimeIdentifier> to each of the .csproj files, which definitely isn't what we want long term.

Might be due to dotnet/sdk#10902, but needs more investigation.

Only publish new version of each OSS Gadget tool if it has changed

After #96 we have MultiExtractor building independently from all of OSS Gadget. But each component of OSS Gadget really operates independently, and to reduce no-op version churn we should only publish new versions where there were changes.

Change Shared.csproj RecursiveExtractor Reference

Change the RecursiveExtractor PackageReference to a ProjectReference within Shared.csproj

Use FileStreams instead of Memory Streams

Arrays in .NET have a limit of 2 billion elements in a single element array.

Calling .ToArray() like here on a memory stream longer than 2GB should (?) thus cause only some of the bytes to be returned (or perhaps an error).

I think we should rewrite this to just parse bytewise off of the end of the memory stream. reading individual chunks into buffers as needed.

Support Greater than ~2GB file entries

Doing this requires rewriting the parser to create a multidimensional array of bytes because individual dimensions are limited to 2 billion. https://docs.microsoft.com/en-us/dotnet/api/system.array.length?view=netcore-3.1

Unreliable Build Test: "Cocoapods_Download_Version_Succeeds"

The build test "Cocoapods_Download_Version_Succeeds" is unreliable. During pipeline runs, the test randomly fails. Re-running the test sometimes passes, sometimes doesn't.

Error:

  X Cocoapods_Download_Version_Succeeds [855ms]
  X Cocoapods_Download_Version_Succeeds (pkg:cocoapods/RandomKit,RandomKit.podspec,1) [855ms]
  Error Message:
   No files were downloaded.
  Stack Trace:
     at Microsoft.CST.OpenSource.Tests.DownloadTests.TestDownload(String purl, String targetFilename, Int32 expectedCount) in D:\a\1\s\src\oss-tests\DownloadTests.cs:line 223
   at Microsoft.CST.OpenSource.Tests.DownloadTests.Cocoapods_Download_Version_Succeeds(String purl, String targetFilename, Int32 expectedCount) in D:\a\1\s\src\oss-tests\DownloadTests.cs:line 28
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.ThreadOperations.ExecuteWithAbortSafety(Action action)

{ "time": "2020-09-05 11:46:10.0155", "level": "Trace", "message": "Removing directory C:\\Users\\scovetta\\AppData\\Local\\Temp\\l3ijv3qp.cwv\\[email protected]" }

That final directory ([email protected]) is removed, but l3ijv3qp.cwv remains.

microsoft / ossgadget Goto Github PK

ossgadget's Introduction

OSS Gadget

Included Tools

Package Sources

Basic Usage

Building from Source

Docker Image

Advanced Usage

Reporting Security Vulnerabilities

Contributing to OSS Gadget

ossgadget's People

Contributors

Stargazers

Watchers

Forkers

ossgadget's Issues

ExtractArchive (Shared.vhdx,True)

ExtractArchive (Shared.vhdx,False)

Create an internal queue to push to.

Process by chunks

Problem

Proposed Solution

Problem

Proposed Solution

Possible implementation

Additional Benefits

Recommend Projects

Recommend Topics

Recommend Org

Jobs