jlhood / github-codebuild-logs Goto Github PK

This would remove some of the noisiness associated when only failed logs need to be accessed.

a few folks used this project prior to the existence of a GitHub action for codebuild, it would be nice to add a readme link (other tools, etc) for awareness.

https://github.com/aws-actions/aws-codebuild-run-build

Thank you @jlhood for creating this application.
We've used in AWS SDK for JavaScript (https://github.com/aws/aws-sdk-js and https://github.com/aws/aws-sdk-js-v3) for the past two years.

Is your feature request related to a problem? Please describe.

CodeBuild has now introduced public builds which allows public to access logs.
The feature provided by this serverless app is now natively supported by CodeBuild.

Describe the solution you'd like

Put github-codebuild-logs on deprecation path. An example could be:

• Update README sharing link to blog post Introducing public builds for AWS CodeBuild, advising visitors to used native option instead.
• Add a required option (say IWantToDeployDespiteNativeSupport) in Application Settings when deploying new version so that future users of the application are aware of the native support.

Describe alternatives you've considered

• Not making any changes: This suggestion is not recommended, as:
  • It's convenient for AWS customers to use the feature natively.
  • AWS customers discovering github-codebuild-logs from blog post or GitHub projects already using the feature may not be knowing about native support.
• Archiving this repository: Archiving can be done at a latter time if users of github-codebuild-logs actually switch to native support. Also, archiving without information about native support would create confusion.

Describe the bug
Deployment of the Serverless Application Repository fails with the following error:

[Error] /Resources/ProcessBuildEventsBuildStatusPermission/Properties/SourceArn/Fn::GetAtt: Resource type AWS::Events::Rule does not support attribute {Arn}. Rollback requested by user.

To Reproduce

• Visit Serverless Application Repository given in README
• Click on "Deploy"
• Enter "CodeBuildProjectName"
• Click on "Deploy"
• Notice that stack fails to get deployed and goes to ROLLBACK_COMPLETE state

Expected behavior
The Cloudformation stack serverlessrepo-github-codebuild-logs gets deployed

Screenshots

CodeBuild allows for a custom Log Group name to be defined for a Project.

Because the template assumes the default Log Group name of /aws/codebuild/${ProjectName} is used, if a custom name was set when configuring the CodeBuild Project, the event processor will throw permissions errors.

One solution to this would be to add an optional parameter that is used, perhaps with a Condition, to build the ProcessBuildEvents policy.

Context

Hi I'm one of the maintainers of torchserve at PyTorch which we comaintain with AWS and we use codebuild and your bot to be able to share with community members issues with their PRs. We can't share the codebuild links directly since they're internal to AWS.

When we run our CI we need to pass on both linux-cpu, linux-gpu and windows so there's a total 3 of comments that need to show and not just 1

https://github.com/pytorch/serve/pulls

Problem

However often our CI is flaky or it takes a few commits for people to get their PR right so we end up with with dozens of comments from a bot which drown out the useful comments like reviews. And we can't set DeletePreviousComment because we need 3 comments to show at a time at a minimum from each of the distributions

Proposed solution

So if we could instead the number of previous comments to be something like 3 or 6 it makes the discussions on github PRs much more readable.

I haven't started digging to see what is causing this yet, but some of the presigned URLs appear to be calculating incorrectly.

ex: mattsb42-meta/pipx-in-pipx#12

While debugging an issue with my Windows CI[1], I ran into two issues with keeping track of what was doing what when.

1. I have two CodeBuild projects acting as CI for that repo (Linux and Windows). The CI report contains no indication of which one the report is for. This can make tracking down issues difficult.
2. Windows builds are a lot slower than Linux. :) I ran into one case where I had a syntax issue that also caused my Linux build to fail (commit A). I updated my branch with a fix for this (commit B), and the build for Linux-commit-B actually completed and had a report back in the PR before the Windows-commit-A build finished. Fortunately, I could easily tell from the logs that it had the issue from commit A, but if I hadn't known what to look for, it would have been confusing to try to track down the real issue.

[1] mattsb42-meta/pipx-in-pipx#17

I am not sure why but the sourceVersion field in the JSON event never starts with pr/.

A sample from my event as output when DEBUG is enabled:
'currentPhase': 'COMPLETED', 'buildStatus': 'SUCCEEDED', 'sourceVersion': '686fcdc44df8358e3382e80379d7abec66f47e38', 'resolvedSourceVersion': '686fcdc44df8358e3382e80379d7abec66f47e38',

Am I a missing something silly, do you have to create the PR in a certain way?

I'm not sure why but this repo is broken.

We are not able to make it work.

The lambda: serverlessrepo-github-codebuild-logs-GetBuildLogs-NsIuFeRMuqgw throw error:

[ERROR] AttributeError: 'NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "/var/task/getbuildlogs.py", line 21, in handler
    log_key = api_event.get('queryStringParameters', 
{}
).get('key')

we are on debug mode but we don't see more logs...

CodeBuild now supports writing logs to S3 in addition to or instead of CloudWatch logs. This app should be updated to work in the case that the CodeBuild project is configured to only write to S3 with CloudWatch logs disabled.

CodeBuild writes the logs to a gzipped S3 file. This app should be able to copy an unzipped version of the file to the app's S3 bucket for PR contributors to view.

Is your feature request related to a problem? Please describe.

• There are no update instructions for projects who're already using github-codebuild-logs
• Installation instructions are in README https://github.com/jlhood/github-codebuild-logs#installation-instructions

Describe the solution you'd like
Add update instructions in README

Describe alternatives you've considered
Deleting application stack, and redeploying a new version

Is your feature request related to a problem? Please describe.
github-codebuild-logs posts multiple comments based on how many times CodeBuild was run for the PR, which depends on how many times additional pushes were done to PR branch after the PR was posted. Example aws/aws-sdk-js-v3#824

Describe the solution you'd like
Add optional HidePreviousComments option which is false by default. When true, it will hide previously posted PR comments as Outdated before posting a new one.

Describe alternatives you've considered

• Manually hide old comments from github-codebuild-logs as outdated, for example aws/aws-sdk-js-v3#976
• Use DeletePreviousComments option, but it deletes the comments instead of hiding them

When a CodeBuild build is triggered by something other than a PR (ex: manually), the sourceVersion field might not be present in the build details.

Because this field is currently retrieved by a getitem call[1], if the key is not found, a KeyError is thrown[2].

[1] https://github.com/jlhood/github-codebuild-logs/blob/master/src/build.py#L27-L32
[2]

[ERROR] KeyError: 'sourceVersion'
Traceback (most recent call last):
  File "/var/task/processbuildevents.py", line 29, in handler
    if not build.is_pr_build():
  File "/var/task/build.py", line 36, in is_pr_build
    return self.get_pr_id() is not None
  File "/var/task/build.py", line 29, in get_pr_id
    matches = re.match(r'^pr\/(\d+)', self._get_build_details()['sourceVersion'])

We have a requirement where every resource including stack should has "RoleType" tag. We have policies that will delete right away if tag is missing. This application deployment is failing because of this. Is it possible to add custom tags option at stack level?

Leading to #13, I was troubleshooting the permissions on my access tokens, and since you cannot modify the permissions on an existing token, I had to create a new token for each permissions attempt.

If the token is set through the SAR console on creation, everything works fine. However, if the token is then changed in Secrets Manager, the Github call fails with Invalid Credentials.
I skimmed the github proxy, and it looks like everything is being done correctly (secret value pulled fresh each time), so I'm not entirely sure why this is failing

I did update the value through the Secrets Manager console, which makes me wonder if some whitespace might be sneaking its way in to the value. I'll do some testing with this to see if I can narrow this down.

Because the PR updater attempts to piggy-back on the CodeBuild GitHub permissions, if permissions are not present, the updater will fail.

It looks like even if the CodeBuild Project was configured through GitHub OAUTH, if the GitHub repo is public, CodeBuild does not obtain an auth token. IMO, this is reasonable and good, because they aren't writing anything, so they don't need one. However, it does mean that for these public repos (the primary purpose of this project's functionality), this project does not appear to consistently work.

hephaestus:pipipxx matt$ aws codebuild batch-get-projects --names pipipxx-linux-py37
{
    "projects": [
        {
            "name": "pipipxx-linux-py37",
            "arn": "arn:aws:codebuild:us-west-2:111222333444:project/pipipxx-linux-py37",
            "source": {
                "type": "GITHUB",
                "location": "https://github.com/mattsb42/pipipxx.git",
                "gitCloneDepth": 1,
                "gitSubmodulesConfig": {
                    "fetchSubmodules": false
                },
                "buildspec": ".codebuild/linux-buildspec.yaml",
                "reportBuildStatus": false,
                "insecureSsl": false
            },
            "secondarySources": [],
            "artifacts": {
                "type": "NO_ARTIFACTS"
            },
            "secondaryArtifacts": [],
            "cache": {
                "type": "NO_CACHE"
            },
            "environment": {
                "type": "LINUX_CONTAINER",
                "image": "aws/codebuild/standard:2.0",
                "computeType": "BUILD_GENERAL1_SMALL",
                "environmentVariables": [
                    {
                        "name": "PYTHON_BUILD_VERSION",
                        "value": "3.7",
                        "type": "PLAINTEXT"
                    }
                ],
                "privilegedMode": false,
                "imagePullCredentialsType": "CODEBUILD"
            },
            "serviceRole": "arn:aws:iam::111222333444:role/service-role/test-codebuild-pipipxx",
            "timeoutInMinutes": 60,
            "queuedTimeoutInMinutes": 480,
            "encryptionKey": "arn:aws:kms:us-west-2:111222333444:alias/aws/s3",
            "tags": [],
            "created": 1558918666.096,
            "lastModified": 1558924935.24,
            "webhook": {
                "url": "https://api.github.com/repos/mattsb42/pipipxx/hooks/112678088",
                "payloadUrl": "xxxxxxxxxxxxxxxx",
                "filterGroups": [
                    [
                        {
                            "type": "EVENT",
                            "pattern": "PUSH, PULL_REQUEST_CREATED, PULL_REQUEST_UPDATED, PULL_REQUEST_REOPENED",
                            "excludeMatchedPattern": false
                        }
                    ]
                ]
            },
            "badge": {
                "badgeEnabled": true,
                "badgeRequestUrl": "https://codebuild.us-west-2.amazonaws.com/badges?uuid=eyJlbmNyeXB0ZWREYXRhIjoiQU5URTJKdnBHQkFwTStnckNjNGlaUGZQNk51V0s1V1JDL0FIUjBERWNmYnNnckM5Vy8vNlZNOWUxWElMLyt3SjlVY2RhSTk3QlByaDZsOEVkZFYzWDVNPSIsIml2UGFyYW1ldGVyU3BlYyI6Ik53bU14aTBqQkFkWmpET2UiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D&branch=master"
            },
            "logsConfig": {
                "cloudWatchLogs": {
                    "status": "ENABLED"
                },
                "s3Logs": {
                    "status": "DISABLED",
                    "encryptionDisabled": false
                }
            }
        }
    ],
    "projectsNotFound": []
}