CloudTrail Trails logging for ALL AWS Regions about prowler

[toniblyx]

Hi @sweharris, I'm working on it and I have fixed the first part with tr '\011' '\012' I missed that part on my tests, thanks! I haven't PR that change yet. I'll do it in a moment.

[sweharris]

Now all emergencies are over, I'm able to get back to this. I just cut'n'pasted that check31() into the current master branch, and it looks good.

Before:

 3.1  Ensure a log metric filter and alarm exist for unauthorized API calls (Scored)
       WARNING! CloudWatch group infra/cloudtrail-1234567890-us-east-1 found with metric filters but no alarms associated

After:

 3.1  Ensure a log metric filter and alarm exist for unauthorized API calls (Scored)
       OK!  CloudWatch group infra/cloudtrail-1234567890-us-east-1 found with metric filter 1234567890_unauthorized_api_calls_metric and alarms set for Unauthorized Operation and Access Denied

[sweharris]

Although there's a theory question... do we want all defined CloudWatch groups to have a alarm set, or just one? At present if there are two trails, one with an alarm, one without then we get an "OK" and a "warning". It may make more sense to have a flag that says if one exists or not. This variation is longer but provides similar details.

check31(){
  # "Ensure a log metric filter and alarm exist for unauthorized API calls (Scored)"
  textTitle "$ID31" "$TITLE31" "SCORED" "LEVEL1"
  CHECK31OK=""
  CHECK31WARN=""
  CLOUDWATCH_GROUP=$($AWSCLI cloudtrail describe-trails $PROFILE_OPT --region $REGION --query 'trailList[*].CloudWatchLogsLogGroupArn' --output text| tr '\011' '\012' | awk -F: '{ print $7 }')
  if [[ $CLOUDWATCH_GROUP ]];then
    for group in $CLOUDWATCH_GROUP; do
      CLOUDWATCH_LOGGROUP_REGION=$($AWSCLI cloudtrail describe-trails $PROFILE_OPT --region $REGION --query 'trailList[*].CloudWatchLogsLogGroupArn' --output text | awk -F: '{ print $4 }')
      #METRICFILTER_SET=$($AWSCLI logs describe-metric-filters --log-group-name $group $PROFILE_OPT --region $CLOUDWATCH_LOGGROUP_REGION --query 'metricFilters' | awk '/UnauthorizedOperation/ || /AccessDenied/ {print $3}')
      METRICFILTER_SET=$($AWSCLI logs describe-metric-filters --log-group-name $group $PROFILE_OPT --region $CLOUDWATCH_LOGGROUP_REGION --output text | grep METRICFILTERS | awk 'BEGIN {IGNORECASE=1}; /UnauthorizedOperation/ || /AccessDenied/ {print $3};')
      if [[ $METRICFILTER_SET ]];then
        for metric in $METRICFILTER_SET; do
          #HAS_ALARM_ASSOCIATED=$($AWSCLI cloudwatch describe-alarms $PROFILE_OPT --region $CLOUDWATCH_LOGGROUP_REGION --query 'MetricAlarms[].MetricName' --output text | awk 'BEGIN {IGNORECASE=1}; /UnauthorizedOperation/ || /AccessDenied/;')
          HAS_ALARM_ASSOCIATED=$($AWSCLI cloudwatch describe-alarms $PROFILE_OPT --region $CLOUDWATCH_LOGGROUP_REGION --query 'MetricAlarms[].MetricName' --output text | grep  $metric)
          if [[ $HAS_ALARM_ASSOCIATED ]];then
            CHECK31OK="$CHECK31OK $group:$metric"
          else
            CHECK31WARN="$CHECK31WARN $group:$metric"
          fi
        done
      else
        CHECK31WARN="$CHECK31WARN $group"
      fi
    done

    if [[ $CHECK31OK ]]; then
      for group in $CHECK31OK; do
        metric=${group#*:}
        group=${group%:*}
        textOK "CloudWatch group $group found with metric filter $metric and alarms set for Unauthorized Operation and Access Denied"
      done
    else
      for group in $CHECK31WARN; do
        case $group in
           *:*) metric=${group#*:}
                group=${group%:*}
                textWarn "CloudWatch group $group found with metric filter $metric but no alarms associated"
                ;;
             *) textWarn "CloudWatch group $group found but no metric filters or alarms associated"
        esac
      done
    fi
  else
    textWarn "No CloudWatch group found for CloudTrail events"
  fi
}

[sweharris]

That the same code logic appears 14 times tells me there's a need for refactorisation of this whole section. Possibly create a __check3 function that takes a few parameters and does the consistent logic. Then check31 would just call __check3 with the right values.

Probably also some caching of results, since you call the same API calls dozens of times (how many times is $AWSCLI cloudtrail describe-trails $PROFILE_OPT --region $REGION --query 'trailList[*].CloudWatchLogsLogGroupArn' --output text called? By my count 27 times.) Caching these type of results will reduce API calls and so increase speed, reduce cost, reduce errors...

This is a pretty large rewrite, and I'm not sure I have the spare time. Fortunately you now have the pattern, so aren't dependent on me as a blocker!

[toniblyx]

Hi @GaboP87, well spotted and thanks for your research on it. What I can do is to search in all regions if specific metric filters and alarms have been set and if found, not show any warning message. What do you think?

[GaboP87]

Hi @toniblyx, I think it would be nice to:

• First: check for the existence of the CloudTrail trail from any AWS Region (it is already done since it would be visible from any AWS Region if you set the trail to log API Calls for the entire AWS Account)

• Second: scan ALL the AWS Regions to check for the existence of the CloudWatch Log Group, the necessary Metrics, Alarms, and SNS Endpoints. If none of them are found in any AWS Region, send the correspondent FAILED (WARNING!) message:

  WARNING! CloudWatch group found but no metric filters or alarms associated in any region

  Otherwise, log the check as PASSED (OK!), and maybe specify in which AWS Region they are found:

  OK! the metric and its associated alarm were found in the eu-central-1 region

Let me know if you understood what I ment with this.

Cheers,
Gabriel

[pchaganti]

👍

[sweharris]

If you look at the definition of a trail then it has a region in the field
eg

arn:aws:logs:REGION:ACCOUNT:log-group:TRAILNAME:*

You should probably parse the region from the output and not just the trail name, and then use that as part of the following calls, so you direct it to the right region

[toniblyx]

Hi @GaboP87 @pchaganti @sweharris thanks for your comments. I think have fixed and tested the issue. Can you give it a look please? If something else is wrong please feel free to reopen the case.

Sorry to address this late :)

[sweharris]

So looking at check31 I can see some issues. These issues probably apply the other sections as well.

If there is more than one trail defined then the CLOUDWATCH_GROUP=$($AWSCLI cloudtrail describe-trails $PROFILE_OPT --region $REGION --query 'trailList[*].CloudWatchLogsLogGroupArn' --output text | awk -F: '{ print $7 }') command may only show the first entry

eg

% $AWSCLI cloudtrail describe-trails $PROFILE_OPT --region $REGION --query 'trailList[*].CloudWatchLogsLogGroupArn' --output text | awk -F: '{print $7}'
infra/cloudtrail-111111111111-us-west-2

% $AWSCLI cloudtrail describe-trails $PROFILE_OPT --region $REGION --query 'trailList[*].CloudWatchLogsLogGroupArn'
[
    "arn:aws:logs:us-west-2:111111111111:log-group:infra/cloudtrail-111111111111-us-west-2:*",
    "arn:aws:logs:us-west-2:111111111111:log-group:CloudTrail/DefaultLogGroup:*"
]

For some reason the output is all on one line, with a TAB separator, rather than multiple line...

% $AWSCLI cloudtrail describe-trails $PROFILE_OPT --region $REGION --query 'trailList[*].CloudWatchLogsLogGroupArn' --output text
arn:aws:logs:us-west-2:111111111111:log-group:infra/cloudtrail-111111111111-us-west-2:* arn:aws:logs:us-west-2:111111111111:log-group:CloudTrail/DefaultLogGroup:*

You may need to tr '\011' '\012' before the awk to make it return both values.
eg

% $AWSCLI cloudtrail describe-trails $PROFILE_OPT --region $REGION --query 'trailList[*].CloudWatchLogsLogGroupArn' --output text | tr '\011' '\012' | awk -F: '{print $7}'
infra/cloudtrail-111111111111-us-west-2
CloudTrail/DefaultLogGroup

I wonder if this is a change in the aws CLI tool or maybe this has been an error all along. Dunno! I'm using

% pip list | grep awscli
awscli (1.11.189)

CLOUDWATCH_LOGGROUP_REGION is being set correctly. Good.

METRICFILTER_SET is set wrong

$AWSCLI logs describe-metric-filters --log-group-name $group $PROFILE_OPT --region $CLOUDWATCH_LOGGROUP_REGION --query 'metricFilters' | awk '/UnauthorizedOperation/ || /AccessDenied/ {print $1" "$2}'
"filterPattern": "{

That's because the output line looks like

        "filterPattern": "{ ($.errorCode = \"*UnauthorizedOperation\") || ($.errorCode = \"AccessDenied*\") }",

The whole JSON for that section looks like

    {
        "filterName": "111111111111_unauthorized_api_calls_metric",
        "metricTransformations": [
            {
                "metricValue": "1",
                "metricNamespace": "CISBenchmark",
                "metricName": "111111111111_unauthorized_api_calls_metric"
            }
        ],
        "creationTime": 1510691180831,
        "filterPattern": "{ ($.errorCode = \"*UnauthorizedOperation\") || ($.errorCode = \"AccessDenied*\") }",
        "logGroupName": "infra/cloudtrail-111111111111-us-west-2"
    },

Because this variable is set wrong, we then fail the HAS_ALARM_ASSOCIATED

If I manually set it right then the thing doesn't look right at all...

HAS_ALARM_ASSOCIATED=$($AWSCLI cloudwatch describe-alarms $PROFILE_OPT --region $CLOUDWATCH_LOGGROUP_REGION --query 'MetricAlarms[].MetricName' --output text | awk 'BEGIN {IGNORECASE=1}; /UnauthorizedOperation/ || /AccessDenied/;')

Wouldn't match the awk statement because you're checking on MetricName and so should be looking for the 111111111111_unauthorized_api_calls_metric value. Alarms don't have details like "accessdenied" in them. This specific alarm looks like

        {
            "EvaluationPeriods": 1,
            "AlarmArn": "arn:aws:cloudwatch:us-west-2:111111111111:alarm:111111111111_unauthorized_api_calls_alarm",
            "StateUpdatedTimestamp": "2017-11-20T18:34:12.602Z",
            "AlarmConfigurationUpdatedTimestamp": "2017-11-14T20:26:21.698Z",
            "ComparisonOperator": "GreaterThanOrEqualToThreshold",
            "AlarmActions": [
                "arn:aws:sns:us-west-2:111111111111:alarm-ops"
            ],
            "Namespace": "CISBenchmark",
            "StateReasonData": "{\"version\":\"1.0\",\"queryDate\":\"2017-11-20T18:34:12.596+0000\",\"statistic\":\"Sum\",\"period\":300,\"recentDatapoints\":[],\"threshold\":1.0}",
            "Period": 300,
            "StateValue": "INSUFFICIENT_DATA",
            "Threshold": 1.0,
            "AlarmName": "111111111111_unauthorized_api_calls_alarm",
            "Dimensions": [],
            "Statistic": "Sum",
            "StateReason": "Insufficient Data: 1 datapoint was unknown.",
            "InsufficientDataActions": [],
            "OKActions": [],
            "ActionsEnabled": true,
            "MetricName": "111111111111_unauthorized_api_calls_metric"
        },

[toniblyx]

Ok @sweharris I get the second part of your comment about METRICFILTER_SET and HAS_ALARM_ASSOCIATED, what do you think is the best way to check it? I think also the way it is checking currently is a bit inconsistent since depends on how you name your alarms. I'd like to add a generic name or filter though. Section 3 is probably the less helpful for me since there are other tools that can handle alarms or alerts in case of API or configuration changes but still useful for a proper security assessment. Thanks.

[sweharris]

For METRICFILTER you might be able to do something like

METRICFILTER_SET=$($AWSCLI logs describe-metric-filters --log-group-name $group $PROFILE_OPT --region $CLOUDWATCH_LOGGROUP_REGION --output text | awk '/UnauthorizedOperation/ || /AccessDenied/ {print $3}')

That assumes the "--output text" format doesn't change. Currently it appears to look like

METRICFILTERS   1510691180831   111111111111_unauthorized_api_calls_metric      { ($.errorCode = "*UnauthorizedOperation") || ($.errorCode = "AccessDenied*") } infra/cloudtrail-111111111111-us-west-2

So the third field is the metric name.

You can then search for that in the alarm output

HAS_ALARM_ASSOCIATED=$($AWSCLI cloudwatch describe-alarms $PROFILE_OPT --region $CLOUDWATCH_LOGGROUP_REGION --query 'MetricAlarms[].MetricName' --output text | grep  $METRICFILTER_SET)

It might work :-) But I'm guessing a lot...

You may need to take into account that multiple metrics might match the search pattern...

[toniblyx]

Ok, I think I get it @sweharris.

check31(){
  ID31="3.1,3.01"
  TITLE31="Ensure a log metric filter and alarm exist for unauthorized API calls (Scored)"
  textTitle "$ID31" "$TITLE31" "SCORED" "LEVEL1"
  CLOUDWATCH_GROUP=$($AWSCLI cloudtrail describe-trails $PROFILE_OPT --region $REGION --query 'trailList[*].CloudWatchLogsLogGroupArn' --output text| tr '\011' '\012' | awk -F: '{ print $7 }')
  if [[ $CLOUDWATCH_GROUP ]];then
    for group in $CLOUDWATCH_GROUP; do
      CLOUDWATCH_LOGGROUP_REGION=$($AWSCLI cloudtrail describe-trails $PROFILE_OPT --region $REGION --query 'trailList[*].CloudWatchLogsLogGroupArn' --output text | awk -F: '{ print $4 }')
      #METRICFILTER_SET=$($AWSCLI logs describe-metric-filters --log-group-name $group $PROFILE_OPT --region $CLOUDWATCH_LOGGROUP_REGION --query 'metricFilters' | awk '/UnauthorizedOperation/ || /AccessDenied/ {print $3}')
      METRICFILTER_SET=$($AWSCLI logs describe-metric-filters --log-group-name $group $PROFILE_OPT --region $CLOUDWATCH_LOGGROUP_REGION --output text | grep METRICFILTERS | awk 'BEGIN {IGNORECASE=1}; /UnauthorizedOperation/ || /AccessDenied/ {print $3};')
      if [[ $METRICFILTER_SET ]];then
        for metric in $METRICFILTER_SET; do
          #HAS_ALARM_ASSOCIATED=$($AWSCLI cloudwatch describe-alarms $PROFILE_OPT --region $CLOUDWATCH_LOGGROUP_REGION --query 'MetricAlarms[].MetricName' --output text | awk 'BEGIN {IGNORECASE=1}; /UnauthorizedOperation/ || /AccessDenied/;')
          HAS_ALARM_ASSOCIATED=$($AWSCLI cloudwatch describe-alarms $PROFILE_OPT --region $CLOUDWATCH_LOGGROUP_REGION --query 'MetricAlarms[].MetricName' --output text | grep  $metric)
          if [[ $HAS_ALARM_ASSOCIATED ]];then
            textOK "CloudWatch group $group found with metric filter $metric and alarms set for Unauthorized Operation and Access Denied"
          else
            textWarn "CloudWatch group $group found with metric filter $metric but no alarms associated"
          fi
        done
      else
        textWarn "CloudWatch group $group found but no metric filters or alarms associated"
      fi
    done
  else
    textWarn "No CloudWatch group found for CloudTrail events"
  fi
}

I have tried even with multiple metrics matching the same and it seems to work. Can you give it a try and let me know? I'd fix the rest of section 3 accordingly. Unless you want to send a PR ;)

Thanks!

[sweharris]

I'm sorry but I've just had a family emergency and am now out of the office until next year. That means I'm unable to test this 'cos I don't have access to the AWS account credentials from home (and rightly so :-)).

The code logic looks correct (if my understanding of AWS structures is right!) but I can't run any tests. Sorry I can't be further help! Thank you for the changes; they look good.

[toniblyx]

Interesting @sweharris, actually these notifications alerts and mostly entire section 3, despite being a best practice, it is something to see case by case, each customer likes different notifications, controls, etc. I'll check your code and try add it soon. Thanks!

[toniblyx]

@sweharris I have added your code for check31 and it works fine in my accounts, thanks! I think check32 to check314 should have similar improvements, any PR would be appreciated ;)

[toniblyx]

Thanks @sweharris I'll give it a try :)