Private GitHub repositories are not recognized with version 3.0.0+ of GitHub provider about terrascan

[williepaul]

Appreciate the bug report @HorizonNet. From the docs, it seems we may also want to add a check for "internal" as well.

[HorizonNet]

@williepaul Probably internal could be offloaded to a separate policy. I spoke about this one with GitHub Support as the documentation is a little bit misleading. internal is only available in a GitHub multi-org with an Enterprise account and makes repositories available in all organizations under this account. If you have a single GitHub organization with an Enterprise account only private and public is available. If you still try to create a repository with internal visibility you get an HTTP 422. With that, most users probably don't care about this one.

BTW, I would be happy to help out on this one (also on my other open issue), but only have little experience with Go or OPA. I found your contribution guide. Do you have additional documentation for new contributors?

[williepaul]

All good points @HorizonNet . We'll likely end up adding a separate rule, as it may be easier to maintain going forward.

Personally, if I'm looking at policy issues, I like to test with the Rego playground as it seems to give better feedback than running opa from the command line IMO. There are some examples on that page, but it might be a little overwhelming at first glance. Our policies are 100% rego with Go Template support.

One tip is that current versions of terrascan have a --config-only flag which when combined with the -o json flag lets you view the json-equivalent of the terraform or k8s YAML input file(s). That config output can be copied directly into the "input" field on the rego playground for testing.

https://play.openpolicyagent.org

Example:
Replace the rego policy with the following:

package accurics

privateRepoEnabled[api.id] {
api := input.github_repository[_]
not api.config.private == true
}

Replace the "INPUT" with the following example, which was taken from https://www.terraform.io/docs/providers/github/r/repository.html and passed through terrascan with command "terrascan scan -t gcp --config-only -o json -f [file with terraform example]". (I also set the "private" value to false):

{
"github_repository": [
{
"id": "github_repository.example",
"name": "example",
"source": "test.rego",
"line": 1,
"type": "github_repository",
"config": {
"description": "My awesome codebase",
"name": "example",
"private": false,
"template": [
{
"owner": "github",
"repository": "terraform-module-template"
}
]
}
}
]
}

Clear out the "DATA" field.

Then, you can easily test by trying different things and hitting evaluate. A successful rule triggered shows up as follows:

{
"privateRepoEnabled": [
"github_repository.example"
]
}

...which is just the rule name, followed by a list of resource IDs which contain the violation.

Hope that helps, feel free to ping me if you have some questions.

Regards,
-Willie

[HorizonNet]

@williepaul Thanks for the guideline. Really helped me a lot. Created a draft PR as I would guess there are still things missing. I tried the change in the playground. There it works as I would expect it, but when I build the Terrascan binary via make build and run the whole thing via

bin/terrascan scan -t gcp -f repositories.tf

where repositories.tf is using visibility set to private, it still shows violations. Not sure why yet, but wanted to see if I'm going into the right directions.

What's probably missing on the PR is a test to verify the change. What's the right spot to look at?

[williepaul]

No problem! From the looks of the change, your policy should work. I think the confusion may be that whenever you run the terrascan binary, it doesn't actually pick up the policies from the current directory structure. On initial run, terrascan clones the repo at ~/.terrascan, and so the policies found there are the ones actually used during the scan. Try overwriting the file under ~/.terrascan/pkg/policies/..., and it should work.

[williepaul]

Oh and also I should have mentioned that there is no need to rebuild for policy updates--you can use the current build installed. You may also specify -p [policy path] to have terrascan pick up the policies in any directory instead of the hidden user home dir.

[HorizonNet]

Ah, good point. When using the p flag and pointing it to the updated policies it works as expected when switching between the different visibility options. Let me make this one ready for review to get feedback on what's still missing.