observeinc / terraform-google-collection

Terraform module which streamlines collection from multiple sources within GCP

Observe Google Collection

This module creates a log sink, pub/sub topic, and pub/sub subscription needed to facilitate the collection of asset inventory records, metrics and logs from GCP for a given project.

This module also creates a Cloud Function to fetch some data through the GCP REST API.

Usage

Here is an example manifest for collecting data from a Google Cloud organization.

After running terraform apply, data should start flowing into Pub/Sub. In the Observe UI, one would set up the GCP app. The info from the terraform output and terraform output -raw service_account_private_key are needed to set up the GCP App pollers.

provider "google" {
  project = "YOUR_PROJECT_ID"
  region  = "YOUR_DEFAULT_REGION"
}

module "observe_gcp_collection" {
  source  = "observeinc/collection/google"
  name    = "observe"

  resource = "projects/YOUR_PROJECT_ID"
}

output "project" {
  description = "The Pub/Sub project of the subcription (to be passed to the Pub/Sub poller)"
  value       = module.observe_gcp_collection.project
}

# To extract correct value - terraform output -json | jq -r '.subscription.value.name' 
output "subscription" {
  description = "The Pub/Sub subscription created by this module (to be passed to the Pub/Sub poller)"
  value       = module.observe_gcp_collection.subscription
}

# To extract properly formatted string - terraform output -json | jq -r '.service_account_private_key.value'
output "service_account_private_key" {
  description = "A service account key to be passed to the pollers for Pub/Sub and Cloud Monitoring"
  value       = base64decode(module.observe_gcp_collection.service_account_key.private_key)
  sensitive   = true
}

Requirements

Name	Version
terraform	>= 0.12.21
google	>= 4.15

Providers

Name	Version
google	4.71.0

Modules

No modules.

Resources

Name	Type
google_cloud_scheduler_job.this	resource
google_cloudfunctions_function.gcs_function	resource
google_cloudfunctions_function.this	resource
google_cloudfunctions_function_iam_member.cloud_scheduler	resource
google_folder_iam_member.cloudfunction	resource
google_logging_folder_sink.this	resource
google_logging_organization_sink.this	resource
google_logging_project_sink.this	resource
google_organization_iam_member.cloudfunction	resource
google_project_iam_member.cloudfunction	resource
google_project_iam_member.poller	resource
google_pubsub_subscription.this	resource
google_pubsub_subscription_iam_member.poller_pubsub	resource
google_pubsub_topic.this	resource
google_pubsub_topic_iam_member.cloudfunction_pubsub	resource
google_pubsub_topic_iam_member.sink_pubsub	resource
google_service_account.cloud_scheduler	resource
google_service_account.cloudfunction	resource
google_service_account.poller	resource
google_service_account_key.poller	resource
google_storage_bucket.this	resource
google_storage_bucket_iam_member.bucket_iam	resource
google_storage_bucket_iam_member.gcs_function_bucket_iam	resource
google_folder.this	data source
google_project.this	data source

Inputs

Name	Description	Type	Default	Required
enable_function	Whether to enable the Cloud function	bool	true	no
folder_include_children	Whether to include all children Projects of a Folder when collecting logs	bool	true	no
function_available_memory_mb	Memory (in MB), available to the function. Default value is 512. Possible values include 128, 256, 512, 1024, etc.	number	512	no
function_bucket	GCS bucket containing the Cloud Function source code	string	"observeinc"	no
function_disable_logging	Whether to disable function logging.	bool	false	no
function_max_instances	The limit on the maximum number of function instances that may coexist at a given time.	number	5	no
function_object	GCS object key of the Cloud Function source code zip file	string	"google-cloud-functions-v0.3.0-alpha.8.zip"	no
function_roles	A list of IAM roles to give the Cloud Function.	set(string)	[ "roles/compute.viewer", "roles/iam.serviceAccountViewer", "roles/cloudscheduler.viewer", "roles/cloudasset.viewer", "roles/browser", "roles/logging.viewer", "roles/monitoring.viewer", "roles/storage.objectCreator", "roles/storage.objectViewer", "roles/storage.objectAdmin", "roles/storage.admin" ]	no
function_schedule_frequency	Cron schedule for the job	string	"0 * * * *"	no
function_timeout	Timeout (in seconds) for the function. Default value is 300 seconds. Cannot be more than 540 seconds.	number	300	no
labels	A map of labels to add to resources (https://cloud.google.com/resource-manager/docs/creating-managing-labels)" Note: Many, but not all, Google Cloud SDK resources support labels.	map(string)	{}	no
logging_exclusions	Log entries that match any of these exclusion filters will not be exported. If a log entry is matched by both logging_filter and one of logging_exclusions it will not be exported. Relevant docs: https://cloud.google.com/logging/docs/reference/v2/rest/v2/billingAccounts.exclusions#LogExclusion	list(object({ name = string description = string filter = string disabled = string }))	[]	no
logging_filter	An advanced logs filter. The only exported log entries are those that are in the resource owning the sink and that match the filter. Relevant docs: https://cloud.google.com/logging/docs/view/building-queries	string	""	no
name	Module name. Used as a name prefix.	string	"observe-collection"	no
poller_roles	A list of IAM roles to give the Observe poller (through the service account key output).	set(string)	[ "roles/monitoring.viewer" ]	no
pubsub_ack_deadline_seconds	Ack deadline for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions)	number	60	no
pubsub_maximum_backoff	Retry policy maximum backoff for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions)	string	"600s"	no
pubsub_message_retention_duration	Message retention for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions)	string	"86400s"	no
pubsub_minimum_backoff	Retry policy minimum backoff for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions)	string	"10s"	no
resource	The identifier of the GCP Resource to monitor. The resource can be a project, folder, or organization. Examples: "projects/my_project-123", "folders/1234567899", "organizations/34739118321"	string	n/a	yes

Outputs

Name	Description
project	The ID of the Project in which resources were created
service_account_key	A service account key to be passed to the pollers for Pub/Sub and Cloud Monitoring
subscription	The Pub/Sub subscription created by this module.
topic	The Pub/Sub topic created by this module.