microsoft / sarif-pattern-matcher

Quality domain agnostic regular expression pattern matcher that persists results to SARIF

sarif-pattern-matcher

Quality domain agnostic regular expression pattern matcher that persists results to SARIF

NuGet packages

The following packages are published from this repository:

	Latest Official Release
Sarif.Pattern.Matcher
Sarif.Pattern.Matcher.Cli
Sarif.Pattern.Matcher.Sdk
Sarif.Pattern.Matcher.Security
RE2.Managed
Strings.Interop

Getting started

• Creating plugins
• Using the client tool
• Using the library

How To Contribute

sarif-pattern-matcher is accepting contributions. If you've submitted a PR for an existing issue, please post a comment in the issue to avoid duplication of effort. See our CONTRIBUTING file for more information - it also contains guidelines for how to submit a PR.

License

"Sarif-pattern-matcher" is licensed under MIT license. View license.

sarif-pattern-matcher's Issues

Create validator StripeKey

Create validator for SquareAccessToken

[AzureFunction] Create unit tests

Please add tests or delete rules for Hockey app and Facebook app credentials

@jameswinkler, @eddynaka

We have no meaningful tests for these checks. Please review these rules for value. And build them out if they should be retained or delete them if they are weak.

Update github pat

https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/

We shouldn't flatten all files to .spam folder

If we do this, we can't easily support plug-ins with validator files that have different dependencies.

We should probably have a top-level directory with the plugin name:

.spam\Security

Release to nuget.org

We should release to nuget.org.

We should change from Microsoft.CodeAnalysis to Microsoft.StaticAnalysis.

@michaelcfanning / @cfaucon, what do you think?

Create validator for SquareOAuthSecret

Copy json files from Security to solution folder

Idea: when we build the solution, our JSON files inside the Security project should be copied to a folder called .spam where the solution exists.

Suggestion: use MSBuild tasks to accomplish this.

Remove 'matchedPattern', when appropriate, from static analysis validator

No longer necessary. Groups["0"] contains this data as a flex match. This is a very intrusive change, whoever picks it up should coordinate with other team members first to make sure the timing is good.

Validate .cer files

@jameswinkler , if we follow what I did in the pfx, we could validate .cer files.

We should create a folder helpers and move the logic that they will share.

ping me if u need help.

cc: @michaelcfanning

Create validator for AmazonMWSAuthToken

Consider update MatchExpression to accept Filtering

Today, every result found by the ContentsRegex is a valid issue.

But we have cases that we know that is a false positive, so it should not trigger.

GoogleGCMServiceAccountValidator casing incorrect and fingerprint is wrong

@eddynaka , @jameswinkler , this is a small fix but high priority

The correct casing of this rule name is GoogleGcmServiceAccountValidator. In .NET, you never case more than two letters in sequence as upper-case.

Also, the fingerprint for this check is broken. Resource is not the right property to use. A 'resource' is a sub-component of a container. So, a database is a resource of a database server.

This rule in general looks weak. What's the insecurity here? If we can't detect an exposure of interest, that can be validated, please just consider deleting this check for now.

Slack web hook rule should break fingerprint into key and id components.

@eddynaka, @jameswinkler, this is high priority.

We currently persist the complete slack web URL as the secret. This prevents us from, say, examining the shannon entropy of the key element of the URL. I think it breaks down like so.

Can we convert the fingerprint to use the id and key elements asap? Thanks!

https://hooks.slack.com/services/{id}/{key}

Create empty projects

Consider update MatchExpression to support multiple regex

Today, each MatchExpression contains a property called ContentsRegex. With that, we are restricted to only one regex per match expression.

But we have cases where a subId might have multiple regexes.

PlugIns build don't move build output path and test results locations

These are emitted within each plugin directory currently. Can we change relevant targets files to redirect to the official build location?

@eddynaka

Create validator for PayPalBraintreeAccessToken

Add (?<fingerprint>) data to new plaintextsecrets.json rules

@eddynaka

Move from Theory to Fact

Check this pr and move from Theory to Fact just like sarif-sdk.

#60

Shouldn't we have a ReturnUnexpectedResponseCode helper?

This would return a consistent message and return value for this scenario.

Consider creating a base class / interface for validators

Today, for each validator, we are just implementing a method. But, sometimes, we need a base configuration across all validators, such as tls1.2 enablement.

Based on that, we should figure out what is the best way to provide this interface / class to facilitate this development.

cc: @cfaucon @michaelcfanning

[Versioning] Check how to create versioning increasing number

Today, we are using nerdbank to generate the versioning.

But when we do that and we publish, it's adding the commit to the versioning, making it hard to check if you are using the latest or not.

Idea: we don't want to create a new branch, for example v1.3.2, update the files and release, since we are doing pre-releases.

Analyzing Google API keys with referer/APK SHA1 cert restrictions should return pass results

Results such as the following aren't necessarily warnings, they may reflect proper security applied to an API key. Generally, it's best not to expose a key but this can be difficult in some web site scenarios or when embedding a key in a mobile app (even in this latter case, though, Google recommends encrypting or obfuscating).

For these use cases, an API key should be restricted, for example, by applying a referer filter or enforcing a check against the mobile app identity. When our validator detects these conditions today, we fire warnings but arguably, these are 'pass' conditions! :)

e:\repros\test.txt(3,1-40): warning SEC101/003: 'test.txt' contains an apparent Google API key, the validity of which could not be determined by runtime analysis (an unexpected exception was caught attempting to validate api key: RequestDenied: API keys with referer restrictions cannot be used with this API).

e:\repros\test.txt(6,1-40): warning SEC101/003: 'test.txt' contains an apparent Google API key, the validity of which could not be determined by runtime analysis (an unexpected exception was caught attempting to validate api key: RequestDenied: This IP, site or mobile application is not authorized to use this API key. Request received from IP address 2601:600:877f:8c60:3033:d967:d038:6ee3, with empty referer).

Improve HTTP client caching/utilization

@cfaucon has explained to us that our current model is not ideal.

@eddynaka

Merge SqlCredential and PSCredential

These two validators are almost identical in test cases and code. The only real difference are the names which can easily be merged with regex.

Merge the two validators.

Create validator for FacebookAccessToken

Create validator for GoogleOAuthKey

Complete and verify dynamic validation for CloudantValidator

While dynamic validation exists for CloudantValidator, we've never verified its accuracy. Testing against a free database https://64de90ff-4b11-4141-b2da-71d8013703be-bluemix.cloudant.com/dashboard.html always yields a 200/successful response, regardless of whether or not credentials are included in a request. It may be that free accounts can't be secured like payed accounts.

If we see that scanning against real data produces successful validation, we can close this work item.

If not, we need to eventually revisit this and complete the implementation. Perhaps the IBM documentation will be improved by then.

Useful links:

• https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-and-populating-a-simple-ibm-cloudant-database-on-ibm-cloud
• https://stackoverflow.com/questions/17035412/access-cloudant-db-using-net-httpclient
• https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-getting-started-with-cloudant
• https://cloud.ibm.com/login - login here, not the page the database link directs you to

[AzureFunction] Add reference to Security project

Add reference to the security project.

When we publish, the rules and the Security.dll should be in the same folder.

Review Sensitive File should return only one result

Today, if we run our test for the ReviewSensitiveFile, you will see that the output sarif is expecting many results because it is validating many times in the method RunMatchExpression.

What I would expect is: if we analyze a file that is an issue, we will just show one result pointing to the file.

What is happening today: it is analyzing all characters inside the file.

cc: @michaelcfanning

New RE2 wrapper todos

• Integrate with BuildRegex.
• Use String8 instead of StringUtf8.
• Consider refactoring names.
• Implement x86 support.
• Error handling, especially when compiling a pattern.
• Test x86 support.
• Ensure that Matches can be used concurrently.
  • [ ]Encoding.UTF8.GetBytes concurrency-safe?

Verify checksum for new GitHub PAT

The new GitHub PAT described here:

https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/

... comes with a checksum in the last 6 characters. Use this in static validation to reduce false positives.

We may be able to calculate CRC32 using existing code in the internal repo, here:
\src\Plugins\Security.Internal\SEC101_102.AdoPatValidator.cs

This library may be useful:

ICSharpCode.SharpZipLib.Checksum;

Search Nuget for "SharpZipLib"

Other useful code:

var crc32 = new Crc32();
byte[] byteArray = Encoding.ASCII.GetBytes(pat);
crc32.Update(byteArray);
string testString = crc32.Value.ToString();

string paddedIntendedChecksum = crc32.Value.ToString().PadLeft(6, '0');

if (paddedIntendedChecksum != checksum)
{
return ValidationState.NoMatch;
}

Review email address regex and create validator using EmailAddress attribute

Security.csproj contains a json that looks for an email using regex. That regex is kinda broken.

Then, after the fix, try to create an email validator.

n

Consider enabling JSON files by default

Talking with @cfaucon , when we analyze data from Cosmos, the data is based on JSON. So, with that in mind, we would need to enable all rules to accept JSON files.

What we propose:

Option 1:

• create a flag in the match expression properties that would enable JSON by default and, if the rule does not apply for JSON files, we would set it to false, for example.

Option 2:

• file vs content detection, for example, pfx vs json.

@michaelcfanning , what do you think?

Create validator for GoogleGCMServiceAccount

Improve Facebook App Credentials validator

... by using the information in this article: https://medium.com/@vibeeshpottayil/how-to-find-the-owner-of-a-facebook-app-id-b3205ebd0049

Author Akamai detection (and validator if feasible)

@eddynaka, this one looks useful, let's make it a priority if we can.

Create validator for GoogleOAuthCredentials

Create validator for this rule: Src/Plugins/Security/SEC101_002.GoogleOAuthCredentialsValidator.cs

Review AssetFingerprint and AssetPlatform enum

Create validator for PicaticKey

Support RE2 memory allocation

Just to let you know, RE2 has an option where we can configure the size of the memory to use.

Right now, it’s a fixed number, but in specific cases, we could change that to something bigger.

😊

Enable CI

Improve NugetCredentials validator/regex

There are an arbitrary number of hosts in a config file, but you can also specify an arbitrary number of user credentials!

and so this helper should return host * secrets fingerprint candidates.

https://docs.microsoft.com/en-us/nuget/reference/nuget-config-file

<packageSourceCredentials>
    <Contoso>
        <add key="Username" value="[email protected]" />
        <add key="Password" value="..." />
        <add key="ValidAuthenticationTypes" value="basic" />
    </Contoso>
    <Test_x0020_Source>
        <add key="Username" value="user" />
        <add key="ClearTextPassword" value="hal+9ooo_da!sY" />
        <add key="ValidAuthenticationTypes" value="basic, negotiate" />
    </Test_x0020_Source>
</packageSourceCredentials>

Complete Google Service Account Key Dynamic Validation

The Google Service Account Key validator attempts to extract and validate application oauth credentials. This is tricky as it requires requesting an access token without knowing the scopes, and with many redirects involved in the flow. Some code is started in https://github.com/microsoft/sarif-pattern-matcher/tree/users/v-jwinkler/GoogleServiceAccountKeyValidator_DynamicValidationAndUnitTests. The code using GoogleWebAuthorizationBroker comes closest, producing output messages like "invalid client id" or "client id deleted", but unfortunately these messages are only visible in a browser that unit tests open. The unit tests themselves hang.

Determine if there's any way to programmatically read the results from the browser, simulate the flow, or find some other technology/library to verify the client id and secret.

Replace AWS package for direct call

#128 (comment)

Improving Documentation

I am facing issues building the solution---running BuildAndTest.cmd from powershell throws a bunch of errors, which seem to highlight missing dependencies. Could you kindly enumerate the dependencies for the project in the documentation? Thanks!

Create validator for TwilioKey