googlecloudplatform / security-analytics

Community Security Analytics provides a set of community-driven audit & threat queries for Google Cloud

Community Security Analytics (CSA)

As organizations go through the Autonomic Security modernization journey, this repository serves as a community-driven list of sample security analytics for auditing cloud usage and for detecting threats to your data & workloads in Google Cloud. These may assist detection engineers, threat hunters and data governance analysts.

CSA is a set of foundational security analytics designed to provide organizations with a rich baseline of pre-built queries and rules that they can readily use to start analyzing their Google Cloud logs including Cloud Audit logs, VPC Flow logs, DNS logs, and more using cloud-native or third-party analytics tools. The source code is provided as is, without warranty. See Copyright & License below.

Current release include:

• YARA-L rules for Google Security Operations
• SQL queries for BigQuery
• SQL queries for Log Analytics

The security use cases below are grouped in 6 categories depending on underlying activity type and log sources:

1. 🚦 Login & Access Patterns
2. 🔑 IAM, Keys & Secrets Admin Activity
3. 🏗️ Cloud Provisoning Activity
4. ☁️ Cloud Workload Usage
5. 💧 Data Usage
6. ⚡ Network Activity

To learn more about the variety of Google Cloud logs, how to enable and natively export these logs to destinations like BigQuery or Google Security Operations for in-depth analytics, refer to Google Cloud Security and access analytics solution guide.

Caution: CSA is not meant to be a comprehensive set of threat detections, but a collection of community-contributed samples to get you started with detective controls. Use CSA in your threat detection and response capabilities (e.g. Security Command Center, Google Security Operations, BigQuery, or third-party SIEM) in conjunction with threat prevention capabilities (e.g. Security Command Center, Cloud Armor, Identity-Aware Proxy and Chrome Enterprise Premium). To learn more about Google’s approach to modern Security Operations, check out the Autonomic Security Operations whitepaper.

Security Analytics Use Cases

#	Cloud Security Threat	Log Source	Audit	Detect	ATT&CK® Techniques
1	🚦 Login & Access Patterns
1.01	Login from a highly-privileged account	Workspace Login Audit (Cloud Identity Logs)		✅	T1078.004
1.02	Suspicious login attempt flagged by Google Workspace	Workspace Login Audit (Cloud Identity Logs)		✅	T1078.004
1.03	Excessive login failures from any user identity	Workspace Login Audit (Cloud Identity Logs)		✅	T1078.004, T1110
1.10	Access attempts violating VPC Service Controls	Audit Logs - Policy	✅	✅	T1078.004, T1537
1.20	Access attempts violating IAP (i.e. BeyondCorp) access controls	HTTP(S) LB Logs	✅	✅
1.30	Cloud Console accesses	Audit Logs - Data Access	✅		T1078.004
2	🔑 IAM, Keys & Secrets Changes
2.02	User added to highly-privileged Google Group	Workspace Admin Audit	✅	✅	T1078.004, T1484.001
2.20	Permissions granted over a Service Account	Audit Logs - Admin Activity	✅	✅	T1484.002
2.21	Permissions granted to impersonate Service Account	Audit Logs - Admin Activity	✅	✅	T1484.002
2.22	Permissions granted to create or manage Service Account keys	Audit Logs - Admin Activity	✅	✅	T1484.002
2.30	Service accounts or keys created by non-approved identity	Audit Logs - Admin Activity	✅	✅	T1136.003
2.40	User access added (or removed) from IAP-protected HTTPS services	Audit Logs - Admin Activity	✅	✅	T1484.002
3	🏗️ Cloud Provisioning Activity
3.01	Changes made to logging settings	Audit Logs - Admin Activity	✅	✅	T1562.008
3.02	Disabling VPC Flows logging	Audit Logs - Admin Activity		✅	T1562.008
3.11	Unusual number of firewall rules modified in the last 7 days	Audit Logs - Admin Activity		✅	T1562.007
3.12	Firewall rules modified or deleted in the last 24 hrs	Audit Logs - Admin Activity	✅	✅	T1562.007
3.13	VPN tunnels created or deleted	Audit Logs - Admin Activity	✅	✅	T1133
3.14	DNS zones modified or deleted	Audit Logs - Admin Activity	✅	✅	T1578
3.15	Cloud Storage buckets modified or deleted by unfamiliar user identities	Audit Logs - Admin Activity	✅	✅	T1578
3.20	VMs deleted in the last 7 days	Audit Logs - Admin Activity	✅		T1578
3.21	Cloud SQL databases created, modified or deleted	Audit Logs - Admin Activity	✅		T1578
4	☁️ Cloud Workload Usage
4.01	Unusually high API usage by any user identity	Audit Logs	✅	✅	T1106
4.10	Autoscaling usage in the past month	Audit Logs - Admin Activity	✅		T1496
4.11	Autoscaling usage per day in the past month	Audit Logs - Admin Activity	✅		T1496
4.20	Resource access by certain user identities in the past month	Audit Logs	✅		T1106
4.21	Resource access by certain user identities in the past month (aggregated by day)	Audit Logs	✅		T1106
4.30	Which users most frequently used LLM models?	Audit Logs - Data Access	✅	✅	T1496, AML.T0051, AML.T0057
4.31	Usage of LLM models over time	Audit Logs - Data Access	✅	✅	T1496, AML.T0051, AML.T0057
5	💧 Data Usage
5.01	Which users most frequently accessed data in the past week?	Audit Logs - Data Access	✅		T1530
5.02	Which users accessed most amount of data in the past week?	Audit Logs - Data Access	✅		T1530
5.03	How much data was accessed by each user per day in the past week?	Audit Logs - Data Access	✅		T1530
5.04	Which users accessed data in a given table in the past month?	Audit Logs - Data Access	✅		T1078.004
5.05	What tables are most frequently accessed and by whom?	Audit Logs - Data Access	✅		T1530
5.06	Top 10 queries against BigQuery in the past week	Audit Logs - Data Access	✅		T1530
5.07	Any queries doing very large scans?	Audit Logs - Data Access	✅	✅	T1530
5.08	Any destructive queries or jobs (i.e. update or delete)?	Audit Logs	✅	✅	T1565.001
5.10	Recent data read with granular access and permissions details	Audit Logs - Data Access	✅		T1074, T1213
5.11	Recent dataset activity with granular permissions details	Audit Logs - Admin Activity	✅		T1074, T1213
5.20	Most common data (and metadata) access actions in the past month	Audit Logs - Data Access	✅	✅	T1530
5.30	Cloud Storage buckets enumerated by unfamiliar user identities	Audit Logs - Data Access	✅	✅	T1530
5.31	Cloud Storage objects accessed from a new IP	Audit Logs - Data Access	✅	✅	T1530
6	⚡ Network Activity
6.01	Hosts reaching out to many other hosts or ports per hour	VPC Flow Logs	✅	✅	T1046
6.10	Connections from a new IP to an in-scope network	VPC Flow Logs	✅	✅	T1018
6.15	List all IP addresses with any associated entities	VPC Flow Logs	✅		T1018, T1046
6.20	Connections blocked by Cloud Armor	HTTP(S) LB Logs	✅	✅	T1071
6.21	Log4j 2 vulnerability exploit attempts	HTTP(S) LB Logs		✅	T1190
6.22	Any remote IP addresses attempting to exploit Log4j 2 vulnerability?	HTTP(S) LB Logs		✅	T1190
6.23	Spring4Shell vulnerability exploit attempts (CVE-2022-22965)	HTTP(S) LB Logs		✅	T1190
6.30	Virus or malware detected by Cloud IDS	Cloud IDS Threat Logs		✅	T1059
6.31	Traffic sessions of high severity threats detected by Cloud IDS	Cloud IDS Threat Logs, Cloud IDS Traffic Logs		✅	T1071
6.40	Top 10 DNS queried domains	Cloud DNS Logs	✅	✅	T1071.004

Dataform for CSA on BigQuery

The dataform folder contains the Dataform repo to automate deployment of CSA queries in BigQuery for optimized performance and cost. Use this Dataform repo to operationalize CSA use cases as reports and alerts powered by BigQuery. This Dataform project deploys and orchestrates pre-built ELT pipelines to filter, normalize and model log data leveraging incremental summary tables, lookup tables and views for fast, cost-effective and simpler querying. See underlying README for more details.

CI/CD for CSA on Google Security Operations

The cicd folder contains a set of scripts to help you with storing CSA YARA-L detection rules as code and testing/deploying updates you and your team make in an automated fashion. Whether you use GitHub Actions, Google Cloud Build or Azure DevOps, you can use the corresponding scripts to automatically test and deploy new or modified rules into your Google Security Operations instance. See underlying README for more details.

Support

This is not an officially supported Google product. Queries, rules and other assets in Community Security Analytics (CSA) are community-supported. Please don't hesitate to open a GitHub issue if you have any question or a feature request.

Contributions are also welcome via Github pull requests if you have fixes or enhancements to source code or docs. Please refer to our Contributing guidelines.

Copyright & License

Copyright 2022 Google LLC

Queries, rules and other assets under Community Security Analytics (CSA) are licensed under the Apache license, v2.0. Details can be found in LICENSE file.

security-analytics's Issues

Terraform module to Set the Export filters in chronicle in GCP.

Is there any terraform- google module that can configure/set the 'Custom export filter settings' in chronicle.
Kindly share any details around this.

[question] How do you use "alert's" views?

How do you use "alert's" views created by dataform to trigger an alert?

Do you have any example that you can share?

CI/CD Readme file points to the wrong folder for Github action workflow

Following document

Suggest that the file should be placed in a .github/workflow, however correct folder should be .github/workflows for Github action pipeline to work properly.

URGENT: Github Actions failing to read SA_CREDENTIAL

Hi Team,

Issue

When running the github actions workflow, it errors out at the pre-upload dry run phase.

When run locally the process works fine, our created rules are ingested into our chronicle rules editor, confirming we are using the correct Google API Key.

However, when we format our Google API key using the cat ~/malachite-abc-7ba40dd4f123.json | tr '\n' ' ' | sed -r 's/\"/\\"/g' (as instructed in https://github.com/GoogleCloudPlatform/security-analytics/blob/main/cicd/README.md) once we insert the resulting formatted key into the repository SA_CREDENTIAL the github action fails.

Screenshot(s)

HELP

Please let us know where we have gone wrong or if there is a bug in the process that's blocking us from creating the CICD.
We will continue working on our end to identify any mistakes we've made too.

Standardize BigQuery backend table formatting for multiple tables

The queries for the BigQuery backend do not have a standardized format for dataset tables with multiple days of data. For example, BigQuery stores multiple days of Cloud Audit Logs in a dataset like the following:

[MY_PROJECT_ID].[MY_DATASET_ID].cloudaudit_googleapis_com_activity_YYYYMMDD
[MY_PROJECT_ID].[MY_DATASET_ID].cloudaudit_googleapis_com_activity_20220916
[MY_PROJECT_ID].[MY_DATASET_ID].cloudaudit_googleapis_com_activity_20220915
…

The queries provided in a majority of the samples only query on a single table, such as 3_01_logging_settings_modified.sql. I believe all the sample queries should accommodate both single day tables and multiple day formats to improve their ease of use. Some queries, like 4_01_unusually_high_api_usage_by_user_identity.sql, accommodate multiple tables already. However the _* suffix formatting used in 4_01 would not accommodate a single day table.

I propose that all queries should pull from a wildcard table to alleviate this issue like the following:

[MY_PROJECT_ID].[MY_DATASET_ID].cloudaudit_googleapis_com_activity*

To prevent the queries from running over massive amounts of data for tables spanning months of logs, we could also include the following delimiter to the queries themselves to allow users to granularly select the time slot for the queries to run.

WHERE 
  _TABLE_SUFFIX = FORMAT("_%s", FORMAT_DATE("%Y%m%d", CURRENT_DATE()))
  AND …

References: https://cloud.google.com/bigquery/docs/querying-wildcard-tables

Incorrect use of operator OR in file

Dear repo maintainers,
while I was investigating the queries for log analytics, I found that there is an error in file:

security-analytics/backends/log_analytics/sql/1_30_console_accesses.sql

Line 34 in 4e139fe

AND proto_payload.audit_log.method_name IN ("GetProject" OR "FindOrCreateOrganization")

comma needs to be used instead of an OR, within parentheses of operator IN.
That way the query succeeds, otherwise syntax error is given.