terraform-google-modules / terraform-google-pubsub

Creates Pub/Sub topic and subscriptions associated with the topic

terraform-google-pubsub

This module makes it easy to create Google Cloud Pub/Sub topic and subscriptions associated with the topic.

Compatibility

This module is meant for use with Terraform 0.13+ and tested using Terraform 1.0+. If you find incompatibilities using Terraform >=0.13, please open an issue. If you haven't upgraded and need a Terraform 0.12.x-compatible version of this module, the last released version intended for Terraform 0.12.x is v1.9.0.

Usage

This is a simple usage of the module. Please see also a simple setup provided in the example directory.

module "pubsub" {
  source  = "terraform-google-modules/pubsub/google"
  version = "~> 6.0"

  topic      = "tf-topic"
  project_id = "my-pubsub-project"
  push_subscriptions = [
    {
      name                       = "push"                                               // required
      ack_deadline_seconds       = 20                                                   // optional
      push_endpoint              = "https://example.com"                                // required
      x-goog-version             = "v1beta1"                                            // optional
      oidc_service_account_email = "[email protected]"                                     // optional
      audience                   = "example"                                            // optional
      expiration_policy          = "1209600s"                                           // optional
      dead_letter_topic          = "projects/my-pubsub-project/topics/example-dl-topic" // optional
      max_delivery_attempts      = 5                                                    // optional
      maximum_backoff            = "600s"                                               // optional
      minimum_backoff            = "300s"                                               // optional
      filter                     = "attributes.domain = \"com\""                        // optional
      enable_message_ordering    = true                                                 // optional
    }
  ]
  pull_subscriptions = [
    {
      name                         = "pull"                                               // required
      ack_deadline_seconds         = 20                                                   // optional
      dead_letter_topic            = "projects/my-pubsub-project/topics/example-dl-topic" // optional
      max_delivery_attempts        = 5                                                    // optional
      maximum_backoff              = "600s"                                               // optional
      minimum_backoff              = "300s"                                               // optional
      filter                       = "attributes.domain = \"com\""                        // optional
      enable_message_ordering      = true                                                 // optional
      service_account              = "[email protected]"          // optional
      enable_exactly_once_delivery = true                                                 // optional
    }
  ]
  bigquery_subscriptions = [
    {
      name                = "bigquery"              // required
      table               = "project.dataset.table" // required
      use_topic_schema    = true                    // optional
      use_table_schema    = false                   // optional
      write_metadata      = false                   // optional
      drop_unknown_fields = false                   // optional
    }
  ]
  cloud_storage_subscriptions = [
    {
      name                     = "cloud-storage"        // required
      bucket                   = "example-bucket"       // required
      filename_prefix          = "log_events_"          // optional
      filename_suffix          = ".avro"                // optional
      filename_datetime_format = "YYYY-MM-DD/hh_mm_ssZ" // optional
      max_duration             = "60s"                  // optional
      max_bytes                = "10000000"             // optional
      output_format            = "avro"                 // optional
      write_metadata           = false                  // optional
    }
  ]
}

Inputs

Name	Description	Type	Default	Required
bigquery_subscriptions	The list of the Bigquery push subscriptions.	list(map(string))	[]	no
cloud_storage_subscriptions	The list of the Cloud Storage push subscriptions.	list(map(string))	[]	no
create_subscriptions	Specify true if you want to create subscriptions.	bool	true	no
create_topic	Specify true if you want to create a topic.	bool	true	no
grant_bigquery_project_roles	Specify true if you want to add bigquery.metadataViewer and bigquery.dataEditor roles to the default Pub/Sub SA.	bool	true	no
grant_token_creator	Specify true if you want to add token creator role to the default Pub/Sub SA.	bool	true	no
message_storage_policy	A map of storage policies. Default - inherit from organization's Resource Location Restriction policy.	map(any)	{}	no
project_id	The project ID to manage the Pub/Sub resources.	string	n/a	yes
pull_subscriptions	The list of the pull subscriptions.	list(map(string))	[]	no
push_subscriptions	The list of the push subscriptions.	list(map(string))	[]	no
schema	Schema for the topic.	object({ name = string type = string definition = string encoding = string })	null	no
subscription_labels	A map of labels to assign to every Pub/Sub subscription.	map(string)	{}	no
topic	The Pub/Sub topic name.	string	n/a	yes
topic_kms_key_name	The resource name of the Cloud KMS CryptoKey to be used to protect access to messages published on this topic.	string	null	no
topic_labels	A map of labels to assign to the Pub/Sub topic.	map(string)	{}	no
topic_message_retention_duration	The minimum duration in seconds to retain a message after it is published to the topic.	string	null	no

Outputs

Name	Description
id	The ID of the Pub/Sub topic
subscription_names	The name list of Pub/Sub subscriptions
subscription_paths	The path list of Pub/Sub subscriptions
topic	The name of the Pub/Sub topic
topic_labels	Labels assigned to the Pub/Sub topic
uri	The URI of the Pub/Sub topic

Requirements

Installation Dependencies

• Terraform >= 0.13.0
• terraform-provider-google plugin >= v2.13

Configure a Service Account

In order to execute this module you must have a Service Account with the following:

Roles

• roles/pubsub.admin

Enable APIs

In order to operate with the Service Account you must activate the following APIs on the project where the Service Account was created:

• Cloud Pub/Sub API

Service Account Credentials

You can pass the service account credentials into this module by setting the following environment variables:

• GOOGLE_CREDENTIALS
• GOOGLE_CLOUD_KEYFILE_JSON
• GCLOUD_KEYFILE_JSON

See more details.

terraform-google-pubsub's Issues

Make token_creator_binding and bigquery_data_editor_binding optional

Please consider making these bindings optional:

terraform-google-pubsub/main.tf

Lines 34 to 46 in 3670eaf

	resource "google_project_iam_member" "bigquery_metadata_viewer_binding" {
	count = length(var.bigquery_subscriptions) != 0 ? 1 : 0
	project = var.project_id
	role = "roles/bigquery.metadataViewer"
	member = "serviceAccount:${local.pubsub_svc_account_email}"
	}
	resource "google_project_iam_member" "bigquery_data_editor_binding" {
	count = length(var.bigquery_subscriptions) != 0 ? 1 : 0
	project = var.project_id
	role = "roles/bigquery.dataEditor"
	member = "serviceAccount:${local.pubsub_svc_account_email}"
	}

Users of the module typically do not have the project IAM permissions and those grants are managed externally to this module.

module does not control retain_acked_messages

I'd create a PR but I understandably dont have push access to add the following code snippet here and here
retain_acked_messages = lookup( var.pull_subscriptions[count.index], "retain_acked_messages", null, )

Use schema already created in a new topic

It's impossible to use a schema already created in a new topic, and that can be done in the GCP console. For example, I have this topic

module "pubsub-1" {
  source              = "[email protected]:XXXXXX/terraform-google-pubsub"
  project_id          = var.project_id
  topic_labels        = local.labels

  topic = "test-topic"
  schema = {
    name       = "test-schema"
    type       = "AVRO"
    encoding   = "JSON"
    definition = "{\n  \"type\" : \"record\",\n  \"name\" : \"Avro\",\n  \"fields\" : [\n    {\n      \"name\" : \"StringField\",\n      \"type\" : \"string\"\n    },\n    {\n      \"name\" : \"IntField\",\n      \"type\" : \"int\"\n    }\n  ]\n}\n"
  }
}

But now if I want to create another topic (similar than this code) with the same schema I try to use the schema created with module "pubsub-1" and there isn't a way to call and old schema, or create exactly the same but only changing the name. The response in this last case doing terraform apply is:
Error: Error creating Schema: googleapi: Error 409: Resource already exists in the project (resource=projects/xxxxx/schemas/test-schema).

It cloud be interesting if there is a way to call by their id or their name another schema already created, via terraform or not. I think schemas are a property that must be shared with different topics, and I find it very inefficient to create a lot of schemas for every topic that I need.

Sorry btw if I'm not using the right terms.

removing dead_letter_topic from subscription results in an error

When you remove the dead_letter_topic from the subscription specification, the module fails to revoke the binding from the service account on the DLQ:

  # module.pubsub["my-topic"].google_pubsub_topic_iam_member.pull_topic_binding[1] must be replaced
-/+ resource "google_pubsub_topic_iam_member" "pull_topic_binding" {
      ~ id      = "projects/my-project/topics/my-second-subscription-dlq/roles/pubsub.publisher/serviceaccount:[email protected]" -> (known after apply)
        member  = "serviceAccount:[email protected]"
        project = "my-project"
        role    = "roles/pubsub.publisher"
      - topic   = "projects/my-project/topics/my-second-subscription-dlq" -> null # forces replacement
    }

The error:

module.pubsub["my-topic"].google_pubsub_topic_iam_member.pull_topic_binding[1]: Creating...

Error: Import id "" doesn't match any of the accepted formats: [projects/(?P[^/]+)/topics/(?P[^/]+) (?P[^/]+)/(?P[^/]+) (?P[^/]+)]

google_pubsub_subscription.push_subscriptions block should not be added if there are no push subscriptions

TL;DR

The google_pubsub_subscription.push_subscriptions is empty if you don't specify any push_subscription, but Terraform reports a diff if you don't specify any.

Expected behavior

The push_config block should not be added to the google_pubsub_subscription.push_subscriptions if no push_subscriptions are there because it's an optional field. Maybe making it dynamic may solve this?

Observed behavior

  # module.whatever.google_pubsub_subscription.push_subscriptions["whatever"] will be updated in-place
  ~ resource "google_pubsub_subscription" "push_subscriptions" {
        id                         = "projects/AAAAAAAAA/subscriptions/BBBBBBBB-subscription"
        name                       = "whatever"
        # (8 unchanged attributes hidden)


      + push_config {
        }
        # (1 unchanged block hidden)
    }

Terraform Configuration

module "whatever_pubsub" {
  project_id = var.google_project_id
  source     = "terraform-google-modules/pubsub/google"
  topic      = "camera-integration"
  version    = "3.0.0"

  push_subscriptions = [
    {
      name          = "whatever"
      push_endpoint = ""
    }
  ]
}

Terraform Version

Terraform v1.0.9
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v3.84.0
+ provider registry.terraform.io/hashicorp/google-beta v3.88.0
+ provider registry.terraform.io/hashicorp/null v2.1.2
+ provider registry.terraform.io/hashicorp/random v2.3.1

### Additional information

_No response_

Support message_retention_duration field in the google_pubsub_topic

TL;DR

The google_pubsub_topic has a message_retention_policy field (https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic#argument-reference) which is currently not supported as an input argument via the terraform-google-pubsub resource (https://registry.terraform.io/modules/terraform-google-modules/pubsub/google/latest).

Terraform Resources

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic

Detailed design

I believe we could just add a line to the pubsub_topic here: https://github.com/terraform-google-modules/terraform-google-pubsub/blob/master/main.tf#L93

Additional information

No response

Unsupported attribute error on "subscription_paths"

TL;DR

When I run terraform plan without changing anything in the source code, I get an error.

Expected behavior

terraform plan should terminates successfully. The pubsub resource has already been created by terraform, so nothing should be changed.

Observed behavior

Error: Unsupported attribute
on .terraform/modules/pubsub.pubsub/outputs.tf line 49, in output "subscription_paths":
 google_pubsub_subscription.pull_subscriptions.*.path,
This object does not have an attribute named "path".

Terraform Configuration

module "pubsub" {
  source  = "terraform-google-modules/pubsub/google"
  version = "~> 1.3"
  topic              = "app"
  project_id         = var.project_id
  pull_subscriptions = [
    {
      name                       = "app"
      ack_deadline_seconds       = "60"
      message_retention_duration = "604800s" 
      retain_acked_messages      = "false"
      enable_message_ordering    = "false"
      dead_letter_policy         = "false"
    },
  ]
}

Terraform Version

docker.io/hashicorp/terraform:1.0.7
hashicorp/terraform:1.0.7
on linux_amd64
hashicorp/google-beta v4.5.0
hashicorp/google v4.5.0
hashicorp/random v3.1.0

Additional information

No response

cleanup config when no dead letter topic present

Pub/Sub hereby refers to the default GCP Pub/Sub SA.

Publisher role resources for Pub/Sub to dead letter topic (DLT) should only be created if a DLT is configured. It should not fallback to the original topic.

terraform-google-pubsub/main.tf

Lines 58 to 80 in 466a9ec

	resource "google_pubsub_topic_iam_member" "push_topic_binding" {
	for_each = var.create_topic ? { for i in var.push_subscriptions : i.name => i } : {}
	project = var.project_id
	topic = lookup(each.value, "dead_letter_topic", "projects/${var.project_id}/topics/${var.topic}")
	role = "roles/pubsub.publisher"
	member = "serviceAccount:${local.pubsub_svc_account_email}"
	depends_on = [
	google_pubsub_topic.topic,
	]
	}
	resource "google_pubsub_topic_iam_member" "pull_topic_binding" {
	for_each = var.create_topic ? { for i in var.pull_subscriptions : i.name => i } : {}
	project = var.project_id
	topic = lookup(each.value, "dead_letter_topic", "projects/${var.project_id}/topics/${var.topic}")
	role = "roles/pubsub.publisher"
	member = "serviceAccount:${local.pubsub_svc_account_email}"
	depends_on = [
	google_pubsub_topic.topic,
	]
	}

Furthermore, subscriber role for Pub/Sub to subscription only needed if the subscription has a DLT configured.

terraform-google-pubsub/main.tf

Lines 82 to 104 in 466a9ec

	resource "google_pubsub_subscription_iam_member" "pull_subscription_binding" {
	for_each = var.create_subscriptions ? { for i in var.pull_subscriptions : i.name => i } : {}
	project = var.project_id
	subscription = each.value.name
	role = "roles/pubsub.subscriber"
	member = "serviceAccount:${local.pubsub_svc_account_email}"
	depends_on = [
	google_pubsub_subscription.pull_subscriptions,
	]
	}
	resource "google_pubsub_subscription_iam_member" "push_subscription_binding" {
	for_each = var.create_subscriptions ? { for i in var.push_subscriptions : i.name => i } : {}
	project = var.project_id
	subscription = each.value.name
	role = "roles/pubsub.subscriber"
	member = "serviceAccount:${local.pubsub_svc_account_email}"
	depends_on = [
	google_pubsub_subscription.push_subscriptions,
	]
	}

tl;dr if there is no DLT configured, none of these should be created.

ref: https://cloud.google.com/pubsub/docs/handling-failures#grant_forwarding_permissions

Add option to control `enable_exactly_once_delivery` for subscriptions.

TL;DR

Currently, enabling exactly once delivery is a preview feature for GCP PubSub subscriptions. I'm curious if y'all are open to adding this flag while it's in a preview state to this module (the official provider supports it).

Terraform Resources

The official pubub_subscription provider supports the enable_exactly_once_delivery.

Detailed design

Implementing this should be a similar change to this one here.

The only major constraint would be that enable_exactly_once_deliver and enable_message_ordering are mutually exclusive. However, handling that doesn't feel like this module's responsibility (the provider should handle it, which it doesn't).

Additional information

No response

Unable to use unique subscription labels

TL;DR

If a topic has multiple subscriptions it would be nice to use different labels for each subscription.

Terraform Resources

Currently the label is applied to all subscriptions of the topic.

terraform-google-pubsub/main.tf

Line 124 in e15c806

labels = var.subscription_labels

Detailed design

Include a map(string) in both the pull_subscription and push_subscription variables.

module "pubsub" {
  source  = "terraform-google-modules/pubsub/google"
  version = "~> 1.8"

  topic      = "tf-topic"
  project_id = "my-pubsub-project"
  push_subscriptions = [
    {
      name                       = "push"
      subscription_label = {my: "label"}
    },
    {
      name                       = "push2"
      subscription_label = {my: "label2"}
    }
  ]
  pull_subscriptions = [
    {
      name                         = "pull" 
      subscription_label           = {my: "label"}                                              
    },
    {
      name                         = "pull2" 
      subscription_label           = {my: "label2"}                                              
    }
  ]
}

Additional information

No response

Add continuous integration

(using Cloud Build)

Failed to query available provider packages Error | Terraform 1.0.8

TL;DR

Hi Team,

Am getting the below error, when running terraform init:

Could not retrieve the list of available versions for provider hashicorp/google: no available releases match the given constraints >= 3.53.0, >= 3.79.0, <= 4.0.0, >= 4.4.0,

when doing terraform init

Am using "terraform-google-modules/pubsub/google" module and the error started coming from today

Below is the code

terraform {
  required_version = ">= 1.0.5, <=1.0.8"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = ">= 3.79.0, <=4.0.0"
    }

}

module "payment-refunds" {
  source  = "terraform-google-modules/pubsub/google"

  topic              = "payment-refunds"
  project_id         = var.project_id
  pull_subscriptions = []

Expected behavior

terraform init should be done successfully

Observed behavior

│ If you wish to specify a provider configuration for the module, add an entry for kubernetes in the required_providers block within the module.
╵

╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider hashicorp/google: no available releases match the given constraints >= 3.53.0, >= 3.79.0, <= 4.0.0, >= 4.4.0, < 5.0.0

Terraform Configuration

terraform {
  required_version = ">= 1.0.5, <=1.0.8"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = ">= 3.79.0, <=4.0.0"
    }

}

module "payment-refunds" {
  source  = "terraform-google-modules/pubsub/google"

  topic              = "payment-refunds"
  project_id         = var.project_id
  pull_subscriptions = []

### Terraform Version

```sh
1.0.8

Additional information

No response

Add types for subscriptions config

TL;DR

Add type to the subscription configs. This helps improve developer experience and ease to use

Terraform Resources

terraform-google-modules/terraform-google-pubsub

Additional information

We have different subscription configs, but currently, they have any type that is hard for new developers to use the module. I had to spend sometimes jump back and forth between the PubSub resource page on the GCP provider Terraform page and GitHub to figure out how to set up BigQuery subscription with other attributes like ack_deadline_seconds, dead_letter_policy, etc..

The BigQuery module has a great table for the input https://github.com/terraform-google-modules/terraform-google-bigquery/blob/master/README.md

Greatly appreciate if we could add something like it

No response

Terraform import returning `no matches found:` even though this exists.

TL;DR

Terraform import returning no matches found: even though this exists.

terraform plan output shows the address of the property: #module.exp_pubsub.google_pubsub_subscription.pull_subscriptions["exp-sub"] will be created

Below is the import command
terraform import module.exp_pubsub.google_pubsub_subscription.pull_subscriptions["exp-sub"] "exp-sub"

Expected behavior

• has moved to *

Observed behavior

terraform import module.exp_pubsub.google_pubsub_subscription.pull_subscriptions["exp-sub"] exp-sub

zsh: no matches found: module.exp_pubsub.google_pubsub_subscription.pull_subscriptions[exp-sub]

Terraform Configuration

old configuration

resource "google_pubsub_topic" "topic" {
  name = "exp-topic"

  labels = {
    app = "experimental"
  }
}

resource "google_pubsub_topic" "topic-dead-letter" {
  name = "exp-topic-dead-letter"

  labels = {
    app = "experimental"
  }
}


resource "google_pubsub_subscription" "sub" {
  name = "exp-sub"
  topic = google_pubsub_topic.topic.name

  labels = {
    app = "experimental"
  }

  ack_deadline_seconds = 10                                                 // optional

  dead_letter_policy {
    dead_letter_topic = google_pubsub_topic.topic-dead-letter.id
  }
}

I want to migrate to managed module, below is the module config

module "exp_pubsub" {
  source  = "terraform-google-modules/pubsub/google"
  version = "~> 4.0"

  topic        = "exp-topic"
  project_id = local.project_id

  pull_subscriptions = [
    {
      name                    = "exp-sub"                                               // required
      ack_deadline_seconds    = 10                                                   // optional
      dead_letter_topic       = google_pubsub_topic.topic-dead-letter.id // optional
    }
  ]

  topic_labels  = {
    app = "experimental"
  }

  subscription_labels = {
    app = "experimental"
  }
}

Terraform Version

4.0.0

Additional information

No response

Add support of `message_storage_policy` to the topic

Issue related to issue #16

Allow creating subscriptions for existing topics

Hello,

Currently, you are not able to create subscriptions if you set create_topic to false. This doesn't let you create subs separately from topics. This is useful when your topic is used by one application and is in a google project that is different from a google project that needs a sub for a different application. This is where is constraint is set for pull subs:

https://github.com/terraform-google-modules/terraform-google-pubsub/blob/master/main.tf#L178

Thanks,

Support Geo Restriction feature

It would be nice to have this feature integrated into this module

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic#example-usage---pubsub-topic-geo-restricted

label support

Support for labeling same as https://www.terraform.io/docs/providers/google/r/pubsub_topic.html

When updating pub sub topic schema subscription are not automatically updated as well

TL;DR

when updating pub sub topic schema the relative subscription are not updated as well.

Expected behavior

No response

Observed behavior

No response

Terraform Configuration

module "pubsub" {
  source  = "terraform-google-modules/pubsub/google"
  version = "~> 5.0"

  topic      = "tf-topic"
  project_id = "my-pubsub-project"
  push_subscriptions = [
    {
      ...
    }


  schema = {
      name = "test"
      type = "PROTOCOL_BUFFER"
      encoding = "JSON"
      definition = file("path to file")

    }
  ]

Terraform Version

1.1.9

Additional information

No response

google_pubsub_subscription_iam_binding overrides custom subscription IAM policies

This module uses google_pubsub_subscription_iam_binding for push and pull subscriptions, and don't allow specifying additional members besides PubSub service account.

https://github.com/terraform-google-modules/terraform-google-pubsub/blob/master/main.tf#L62
https://github.com/terraform-google-modules/terraform-google-pubsub/blob/master/main.tf#L75

Either use google_pubsub_subscription_iam_member or allow to specify additional identities to add to google_pubsub_subscription_iam_binding

Current design overrides my IAM policies for subscriptions every terraform apply to just PubSub service account.

module "pubsub_topic" {
  source  = "terraform-google-modules/pubsub/google"
  version = "~> 1.4"

  topic      = "pubsub-topic"
  project_id = var.project_id
  pull_subscriptions = [
    {
      name = "pull-sub"
    }
  ]
}

resource "google_pubsub_subscription_iam_member" "pull_subscriber_iam" {
  project      = var.project_id
  subscription = "pull-sub"
  role         = "roles/pubsub.subscriber"
  member       = "serviceAccount:dataflow@project_id.iam.gserviceaccount.com"
}

module.pubsub_topic.google_pubsub_subscription_iam_binding.pull_subscription_binding[0] will be updated in-place
   ~ resource "google_pubsub_subscription_iam_binding" "pull_subscription_binding" {
         etag         = "etag"
         id           = "projects/project_id/subscriptions/pull-sub/roles/pubsub.subscriber"
       ~ members      = [
           - "serviceAccount:dataflow@project_id.iam.gserviceaccount.com",
             "serviceAccount:[email protected]",
         ]
         project      = "project_id"
         role         = "roles/pubsub.subscriber"
         subscription = "pull-sub"
    }

Recreating of push_topic_binding every time on version 2

It seems like the resource "google_pubsub_topic_iam_member" "push_topic_binding" is being recreated needlessly every time because of "unknown" member.

  # module.google_pubsub["REMOVED"].google_pubsub_topic_iam_member.push_topic_binding["REMOVED"] must be replaced
-/+ resource "google_pubsub_topic_iam_member" "push_topic_binding" {
      ~ etag    = "REMOVED" -> (known after apply)
      ~ id      = "REMOVED" -> (known after apply)
      ~ member  = "serviceAccount:[email protected]" -> (known after apply) # forces replacement
        # (3 unchanged attributes hidden)
    }

The pubsub topics and everything was set up in a new project, so there is no lingering old state that should cause this. I am not an expert in terraform, but it seems to be related to the way the pubsub_svc_account_email is created from the data block:
pubsub_svc_account_email = "service-${data.google_project.project.number}@gcp-sa-pubsub.iam.gserviceaccount.com"

Here is my setup as well for reference in case I have done something wrong that might cause this:

module "google_pubsub" {
    depends_on = [
        module.project_setup,
        google_project_iam_member.pubsub_service_account_roles,
        module.empower_platform_cloud_build_setup,
        module.cloud_run]
    for_each = local.pubsub_topics_and_subscriptions

    source = "terraform-google-modules/pubsub/google"
    version = "~> 2.0"

    topic = each.key
    create_topic = each.value.create_topic
    project_id = var.project

    push_subscriptions = [for subscription in each.value.subscriptions: {
        name = "subscription-${each.key}-${subscription.service_name}"
        oidc_service_account_email = google_service_account.pubsub_subscription_service_account.email
        ack_deadline_seconds = 200
        push_endpoint = "${lookup(module.cloud_run.deployed_cloud_run_services, subscription.service_name).url}${subscription.root_path}/pubsub/${each.key}"
        enable_message_ordering = true
    }]
}

Removing one subscription affect recreating anothers with data loss.

Subscriptions are defined in the list variable.
Since numerical indexes are used for defining google_pubsub_subscription.push_subscriptions resources, removing one subscription at the beginning of the list makes shift indexes of other subscriptions, so terraform will destroy/create all of them.

Terraform will perform the following actions:

  # google_pubsub_subscription.push_subscriptions[0] must be replaced
-/+ resource "google_pubsub_subscription" "push_subscriptions" {
 ...

Recreating subscription brings the loss of unacked messages and usually unacceptable in most cases.

I'm going to make PR to fix this issue, but this is a breaking change, so it would be great if the fix will be included in the new major release. Please don't release 2.0.0 until this bug will be fixed.

Can not use dynamic service account

TL;DR

Cannot use dynamic value of service account in service_account of pull_configuration. It needs to be static.

Expected behavior

Allow the use dynamic value on service_account of pull_configuration.

Observed behavior

Got following error:

341: for_each = var.create_subscriptions ? { for i in var.pull_subscriptions : i.name => i if lookup(i, "service_account", null) != null } : {}

  var.create_subscriptions is true
  var.pull_subscriptions is list of map of string with 1 element

The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will identify the instances of this resource.

When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.

Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully converge.

Terraform Configuration

module "using_this module" {
  // ...
  pull_subscriptions = [
    {
      name                 = var.some_name
      service_account      = google_service_account.default
    }
  ]
}

Terraform Version

1.3.6

Additional information

I was think we could introduce another key like create_iam

module "using_this module" {
  // ...
  pull_subscriptions = [
    {
      name                 = var.some_name
      create_iam           = true
      service_account      = google_service_account.default
    }
  ]
}

And update like this:
Current:

terraform-google-pubsub/main.tf

Lines 340 to 362 in 3670eaf

	resource "google_pubsub_subscription_iam_member" "pull_subscription_sa_binding_subscriber" {
	for_each = var.create_subscriptions ? { for i in var.pull_subscriptions : i.name => i if lookup(i, "service_account", null) != null } : {}
	project = var.project_id
	subscription = each.value.name
	role = "roles/pubsub.subscriber"
	member = "serviceAccount:${each.value.service_account}"
	depends_on = [
	google_pubsub_subscription.pull_subscriptions,
	]
	}
	resource "google_pubsub_subscription_iam_member" "pull_subscription_sa_binding_viewer" {
	for_each = var.create_subscriptions ? { for i in var.pull_subscriptions : i.name => i if lookup(i, "service_account", null) != null } : {}
	project = var.project_id
	subscription = each.value.name
	role = "roles/pubsub.viewer"
	member = "serviceAccount:${each.value.service_account}"
	depends_on = [
	google_pubsub_subscription.pull_subscriptions,
	]
	}

Proposed:

resource "google_pubsub_subscription_iam_member" "pull_subscription_sa_binding_subscriber" {
  for_each = var.create_subscriptions ? { for i in var.pull_subscriptions : i.name => i if lookup(i, "create_iam", false) != false } : {}

  project      = var.project_id
  subscription = each.value.name
  role         = "roles/pubsub.subscriber"
  member       = "serviceAccount:${each.value.service_account}"
  depends_on = [
    google_pubsub_subscription.pull_subscriptions,
  ]
}

resource "google_pubsub_subscription_iam_member" "pull_subscription_sa_binding_viewer" {
  for_each = var.create_subscriptions ? { for i in var.pull_subscriptions : i.name => i if lookup(i, "create_iam", false) != false } : {}

  project      = var.project_id
  subscription = each.value.name
  role         = "roles/pubsub.viewer"
  member       = "serviceAccount:${each.value.service_account}"
  depends_on = [
    google_pubsub_subscription.pull_subscriptions,
  ]
}

If this is okay, I can work on the PR. Also, happy to learn if you have other suggestions.

Use for_each for subscriptions

Unable to run the tests

I was getting errors while trying to run the tests to check an enhancement I was working on.

make docker_test_prepare
docker run --rm -it \
                -e SERVICE_ACCOUNT_JSON \
                -e TF_VAR_org_id \
                -e TF_VAR_folder_id \
                -e TF_VAR_billing_account \
                -v "${LOCAL_PATH}/public-modules/terraform-google-pubsub":/workspace \
                gcr.io/cloud-foundation-cicd/cft/developer-tools:0 \
                /usr/local/bin/execute_with_credentials.sh prepare_environment
Activated service account credentials for: [[email protected]]
Activated service account credentials for: [[email protected]]
Initializing modules...

Error: Unsupported Terraform Core version

  on versions.tf line 18, in terraform:
  18:   required_version = "~> 0.12.0"

This configuration does not support Terraform version 0.13.5. To proceed,
either choose another supported Terraform version or update this version
constraint. Version constraints are normally set for good reason, so updating
the constraint may lead to other errors or unexpected behavior.


Error: Unsupported Terraform Core version

  on .terraform/modules/project-ci-int-pubsub/versions.tf line 18, in terraform:
  18:   required_version = "~> 0.12.6"

Module module.project-ci-int-pubsub (from
terraform-google-modules/project-factory/google) does not support Terraform
version 0.13.5. To proceed, either choose another supported Terraform version
or update this version constraint. Version constraints are normally set for
good reason, so updating the constraint may lead to other errors or unexpected
behavior.


Error: Unsupported Terraform Core version

  on .terraform/modules/project-ci-int-pubsub/modules/budget/versions.tf line 18, in terraform:
  18:   required_version = "~> 0.12.6"

Module module.project-ci-int-pubsub.module.budget (from ./modules/budget) does
not support Terraform version 0.13.5. To proceed, either choose another
supported Terraform version or update this version constraint. Version
constraints are normally set for good reason, so updating the constraint may
lead to other errors or unexpected behavior.


Error: Unsupported Terraform Core version

  on .terraform/modules/project-ci-int-pubsub/modules/core_project_factory/versions.tf line 18, in terraform:
  18:   required_version = "~> 0.12.6"

Module module.project-ci-int-pubsub.module.project-factory (from
./modules/core_project_factory) does not support Terraform version 0.13.5. To
proceed, either choose another supported Terraform version or update this
version constraint. Version constraints are normally set for good reason, so
updating the constraint may lead to other errors or unexpected behavior.


Error: Unsupported Terraform Core version

  on .terraform/modules/project-ci-int-pubsub/modules/gsuite_group/versions.tf line 18, in terraform:
  18:   required_version = "~> 0.12.6"

Module module.project-ci-int-pubsub.module.gsuite_group (from
./modules/gsuite_group) does not support Terraform version 0.13.5. To proceed,
either choose another supported Terraform version or update this version
constraint. Version constraints are normally set for good reason, so updating
the constraint may lead to other errors or unexpected behavior.


Error: Unsupported Terraform Core version

  on .terraform/modules/project-ci-int-pubsub/modules/project_services/versions.tf line 18, in terraform:
  18:   required_version = "~> 0.12.6"

Module
module.project-ci-int-pubsub.module.project-factory.module.project_services
(from ../project_services) does not support Terraform version 0.13.5. To
proceed, either choose another supported Terraform version or update this
version constraint. Version constraints are normally set for good reason, so
updating the constraint may lead to other errors or unexpected behavior.

make: *** [docker_test_prepare] Error 1

Can't set a custom encryption key for a topic

There is no setting for kms_key_name so all topics created using this module can only use the Google managed keys.

bigquery.dataEditor role assigned only on the dataset specified in inputs

TL;DR

For the bigquery subscription of a topic, the role assign to the gcp pubsub service account is too much permissive.

Terraform Resources

https://registry.terraform.io/modules/terraform-google-modules/pubsub/google/latest

Detailed design

Following the least privileged principle, the permission assigned for the gcp pubsub service account needed for the bigquery subscription to work on a table specified in input is too much permissive.

main.tf#L41
It should not be a google_project_iam_member but more a google_bigquery_dataset_iam_member resource used with the dataset of the subscription as dataset_id

I try to segregate service account by "application initiative" to have a more granular overview of my gcp project and more control about which identity access what

Additional information

The version of this module i used is the "~> 5.0"
The pubsub topic was already created on GCP as well as the subscription and I want to import it on Terraform for single source of truth

Many thanks !

Add expiration_policy

Let's include subscription expiration policy:

https://www.terraform.io/docs/providers/google/r/pubsub_subscription.html#expiration_policy

Error when trying to associate BigQuery push with Pub/Sub in a BigQuery dataset table without the data field created

TL;DR

I receive the error below when using the module with BigQuery example using a dataset with a table without a data field

Expected behavior

I want to join a table from a BigQuery dataset without the data field.

Observed behavior

It is only possible to associate BigQuery with Pub/Sub if the dataset table has the data field.

Terraform Configuration

table = "project:example_dataset.example_table"

Terraform Version

5.27.0

Additional information

N/A

Why is the gcp-sa-pubsub service account made publisher and subscriber by default?

Why is the service-${data.google_project.project.number}@gcp-sa-pubsub.iam.gserviceaccount.com made publisher and subscriber by default?

This does not seem to be in line with the principle of least privilege. An improvement would be to make this optional.

When destroying, get "The given key does not identify an element in this collection value."

When I queue a plan destroy, I obtain this error because of concerned resources are already destroyed, so index 0 does not exist and the output fail…

Terraform version : >= 0.12

Given code :

module "pubsub" {
  source     = "terraform-google-modules/pubsub/google"
  version    = "~> 1.0"
  project_id = var.project_id
  topic      = var.topic_name
  topic_labels = {
    creator = var.label
  }
}

Error: Invalid index

  on .terraform/modules/automatic-dns.pubsub/terraform-google-modules-terraform-google-pubsub-eed95cd/outputs.tf line 18, in output "topic":
  18:   value       = google_pubsub_topic.topic.0.name
    |----------------
    | google_pubsub_topic.topic is empty tuple

The given key does not identify an element in this collection value.


Error: Invalid index

  on .terraform/modules/automatic-dns.pubsub/terraform-google-modules-terraform-google-pubsub-eed95cd/outputs.tf line 23, in output "topic_labels":
  23:   value       = google_pubsub_topic.topic.0.labels
    |----------------
    | google_pubsub_topic.topic is empty tuple

The given key does not identify an element in this collection value.


Error: Invalid index

  on .terraform/modules/automatic-dns.pubsub/terraform-google-modules-terraform-google-pubsub-eed95cd/outputs.tf line 28, in output "id":
  28:   value       = google_pubsub_topic.topic.0.id
    |----------------
    | google_pubsub_topic.topic is empty tuple

The given key does not identify an element in this collection value.


Error: Invalid index

  on .terraform/modules/automatic-dns.pubsub/terraform-google-modules-terraform-google-pubsub-eed95cd/outputs.tf line 33, in output "uri":
  33:   value       = "pubsub.googleapis.com/${google_pubsub_topic.topic.0.id}"
    |----------------
    | google_pubsub_topic.topic is empty tuple

The given key does not identify an element in this collection value.


Error: Invalid index

  on .terraform/modules/backup.backup_topic/terraform-google-modules-terraform-google-pubsub-eed95cd/outputs.tf line 18, in output "topic":
  18:   value       = google_pubsub_topic.topic.0.name
    |----------------
    | google_pubsub_topic.topic is empty tuple

The given key does not identify an element in this collection value.


Error: Invalid index

  on .terraform/modules/backup.backup_topic/terraform-google-modules-terraform-google-pubsub-eed95cd/outputs.tf line 23, in output "topic_labels":
  23:   value       = google_pubsub_topic.topic.0.labels
    |----------------
    | google_pubsub_topic.topic is empty tuple

The given key does not identify an element in this collection value.


Error: Invalid index

  on .terraform/modules/backup.backup_topic/terraform-google-modules-terraform-google-pubsub-eed95cd/outputs.tf line 28, in output "id":
  28:   value       = google_pubsub_topic.topic.0.id
    |----------------
    | google_pubsub_topic.topic is empty tuple

The given key does not identify an element in this collection value.


Error: Invalid index

  on .terraform/modules/backup.backup_topic/terraform-google-modules-terraform-google-pubsub-eed95cd/outputs.tf line 33, in output "uri":
  33:   value       = "pubsub.googleapis.com/${google_pubsub_topic.topic.0.id}"
    |----------------
    | google_pubsub_topic.topic is empty tuple

The given key does not identify an element in this collection value.


Error: Invalid index

  on .terraform/modules/backup.purge_topic/terraform-google-modules-terraform-google-pubsub-eed95cd/outputs.tf line 18, in output "topic":
  18:   value       = google_pubsub_topic.topic.0.name
    |----------------
    | google_pubsub_topic.topic is empty tuple

The given key does not identify an element in this collection value.


Error: Invalid index

  on .terraform/modules/backup.purge_topic/terraform-google-modules-terraform-google-pubsub-eed95cd/outputs.tf line 23, in output "topic_labels":
  23:   value       = google_pubsub_topic.topic.0.labels
    |----------------
    | google_pubsub_topic.topic is empty tuple

The given key does not identify an element in this collection value.


Error: Invalid index

  on .terraform/modules/backup.purge_topic/terraform-google-modules-terraform-google-pubsub-eed95cd/outputs.tf line 28, in output "id":
  28:   value       = google_pubsub_topic.topic.0.id
    |----------------
    | google_pubsub_topic.topic is empty tuple

The given key does not identify an element in this collection value.


Error: Invalid index

  on .terraform/modules/backup.purge_topic/terraform-google-modules-terraform-google-pubsub-eed95cd/outputs.tf line 33, in output "uri":
  33:   value       = "pubsub.googleapis.com/${google_pubsub_topic.topic.0.id}"
    |----------------
    | google_pubsub_topic.topic is empty tuple

The given key does not identify an element in this collection value.

terraform-google-pubsub/outputs.tf

Line 18 in eed95cd

value = google_pubsub_topic.topic.0.name

unable to set expiration_policy.ttl to empty string

TL;DR

In order to prevent a subscription from expiring, the TTL on the subscriptions expiration policy must be set to empty string (if set to null or not provided, then a default 31d expiration is used). However, when setting empty string on the expiration_policy field, the module sets null on the TTL field resulting in default 31d expiration instead of empty string.

Expected behavior

Empty string is set on the subscription.expiration_policy.ttl field.

Observed behavior

null is set on the subscription.expiration_policy.ttl field.

Terraform Configuration

module "mypubsub" {
  source     = "terraform-google-modules/pubsub/google"
  version    = "~> 6.0"
  project_id = var.project_id
  topic      = "mypubsub"
  push_subscriptions = [
    {
      name                       = "mysubscription"
      push_endpoint              = local.url
      x-goog-version             = "v1"
      oidc_service_account_email = local.identity
      expiration_policy          = ""
    }
  ]
}

Terraform Version

1.5

Additional information

I took a look at the module source code in this repo and don't understand why the empty string is turning into a null value. Something unexpected is happening around here:

terraform-google-pubsub/main.tf

Lines 197 to 203 in c496f56

	dynamic "expiration_policy" {
	// check if the 'expiration_policy' key exists, if yes, return a list containing it.
	for_each = contains(keys(each.value), "expiration_policy") ? [each.value.expiration_policy] : []
	content {
	ttl = expiration_policy.value
	}
	}

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): Update go modules and/or dev-tools (github.com/GoogleCloudPlatform/cloud-foundation-toolkit/infra/blueprint-test, go)

Detected dependencies

gomod

test/integration/go.mod

• go 1.22
• go 1.22.4
• github.com/GoogleCloudPlatform/cloud-foundation-toolkit/infra/blueprint-test v0.15.1
• github.com/stretchr/testify v1.9.0

regex

Makefile

• cft/developer-tools 1.21

build/int.cloudbuild.yaml

• cft/developer-tools 1.21

build/lint.cloudbuild.yaml

• cft/developer-tools 1.21

terraform

examples/bigquery/main.tf

• terraform-google-modules/pubsub/google ~> 6.0

examples/cloud_storage/main.tf

• terraform-google-modules/pubsub/google ~> 6.0

examples/kms/main.tf

• terraform-google-modules/pubsub/google ~> 6.0

examples/simple/main.tf

• terraform-google-modules/pubsub/google ~> 6.0

examples/subscriptions_only/main.tf

• terraform-google-modules/pubsub/google ~> 6.0

test/setup/main.tf

• terraform-google-modules/project-factory/google ~> 15.0

test/setup/versions.tf

• google >= 5.31
• google-beta >= 5.31
• hashicorp/terraform >= 0.13

versions.tf

• google >= 5.31, < 6
• hashicorp/terraform >= 0.13

---

• Check this box to trigger a request for Renovate to run again on this repository

When removing the dead_letter_topic variable the IAM permission is not revoked properly

When you remove the dead_letter_topic from the subscription specification, it module fails to revoke the binding from the service account on the DLQ:

  # module.pubsub["my-topic"].google_pubsub_topic_iam_member.pull_topic_binding[1] must be replaced
-/+ resource "google_pubsub_topic_iam_member" "pull_topic_binding" {
      ~ id      = "projects/my-project/topics/my-second-subscription-dlq/roles/pubsub.publisher/serviceaccount:[email protected]" -> (known after apply)
        member  = "serviceAccount:[email protected]"
        project = "my-project"
        role    = "roles/pubsub.publisher"
      - topic   = "projects/my-project/topics/my-second-subscription-dlq" -> null # forces replacement
    }

`enable_message_ordering` option is not available for push subscriptions although it's available through the console

TL;DR

The attribute enable_message_ordering is available through console, and terraform provider for push subscription, but not on this module.
It should be added.

Expected behavior

We should specify enable_message_ordering option for push subscription.

Observed behavior

It's not available.

Terraform Configuration

module "pubsub_topic" {
  source             = "terraform-google-modules/pubsub/google"
  version            = "5.0.0"
  topic              = "topic"
  project_id         = var.gcp_project_id
  push_subscriptions = [
    {
      name = "subscription"
      push_endpoint = "https://example-endpoint.com"
      enable_message_ordering = true
    }
  ]
}

Terraform Version

terraform v1.1.5
module pubsub : v5.0.0
terraform provider google : 4.32.0

Additional information

N/A

Add support for Terraform 0.12

Add filters support for subscription

Can we have support for this..This is a beta feature but still very useful.
https://cloud.google.com/pubsub/docs/filtering

add retry

Perhaps retry helper could be used to deal with this transient issue on new projects:

https://www.terraform.io/docs/extend/resources/retries-and-customizable-timeouts.html#retry

Error: Error creating Topic: googleapi: Error 400: The topic projects/tf-test4-0aa6/topics/repo-events cannot be created or updated due to an organization policy specified at the project or organization level. The organization policy does not allow message storage in any GCP region where Cloud Pub/Sub is present. The organization policy may be updated by your project or organization administrator. See https://console.cloud.google.com/iam-admin/orgpolicies/gcp-resourceLocations?project=tf-test4-0aa6. If the topic's project was recently created, you may need to wait a few minutes for the project's organization policy to be properly initialized, and then retry this operation.

Applying the same main.tf a moment later worked

add option for authoritative pubsub iam resources

TL;DR

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic_iam

Currently we can only create _iam_member resources which are non-authoritative and while that is safe and good in some situations, for single tenant pubsub resources it is preferrable to use authoritative iam resources to ensure that access is fully managed as code.

Terraform Resources

pubsub iam resources

Detailed design

Leverage https://github.com/terraform-google-modules/terraform-google-iam/tree/master/modules/pubsub_subscriptions_iam and https://github.com/terraform-google-modules/terraform-google-iam/tree/master/modules/pubsub_topics_iam

Additional information

tbd

Push Subscription is missing the NoWrapper feature available in the pubsub_subscription resource

Request

The Push Subscription's Push block is missing the NoWrapper feature. This should be added with the additional write metadata flag to support the NoWrapper feature completely.

Terraform Resources

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_subscription is where the no_wrapper feature is exposed.

This is the link to the no_wrapper field https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_subscription#nested_no_wrapper

Detailed design

The Push Subscription is missing the no_wrapper (https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_subscription#no_wrapper) flag/feature.

This no_wrapper and write_metadata fields should be added in the push subscription block https://github.com/terraform-google-modules/terraform-google-pubsub/blob/834f20481a6a634af91aed713e58b02ee6490dd9/main.tf#L221 

This should be optional fields to enable no_wrapper and no_wrapper_write_metadata both defaulting to false

Additional information

This supports use cases when publishing from GCP into another system not honoring the GCP Message Wrapper and when you add additional things into the HTTP Push Request to the Subscribed Endpoint.

breaking feat: set grant_token_creator to false by default

TL;DR

well put here #48 (comment)

or remove it completely, quite the foot gun and probably should not be managed in this module at all

though it will be necessary that the resource be removed from state prior to this change so that it is not deleted, this should be in all caps in the upgrade doc

Terraform Resources

No response

Detailed design

No response

Additional information

No response

Guidance on setting topic-level IAM

TL;DR

Hi I'm not filing a bug or request at the moment but just looking for guidance.

I'm looking to move from the Cloud Foundation Fabric module for pubsub to this module. For the iam input alternative, I don't see this exposed directly in the module, rather just allowing the pubsub P4SA to publish to the topic (in case the serviceAgent role isn't set at the project level, I'm assuming).

What is the expected course of action to set roles/pubsub.publisher at the topic level using this module? Should we just a pubsub_topic_iam_member/binding resource adjacent to this module? Or is there a better practice that I'm missing here? Is there any reason/philosophy as to why this is not supported within the module?

Terraform Resources

No response

Detailed design

No response

Additional information

No response

call multiple modules, or support multiple topics/subscriptions

This module seems to only support one invocation, as calling it twice throws an error about duplicate modules. If you needed to create multiple topics, each with a set of subscriptions, is that not supported? sorry I'm kinda new to TF.

Using version 1.8

When destroying, get "The given key does not identify an element in this collection value." on subscription_names

Hello, I am having the same issue as here: #16

but with regard to .subscription_names

I have seen the fix has not been applied to subscription_names: https://github.com/terraform-google-modules/terraform-google-pubsub/pull/17/files

Thanks for your help

Problem with subscription labels

Hi,
I have a problem with the subscription labels. If I pass a map(string) as required by the variable, I receive the following error for both push and pull subscription:

The given value is not suitable for child module variable "pull_subscriptions"
defined at modules\pubsub\variables.tf:45,1-30: element 0: element "labels":
string required.

Instead, if I pass a string (following the error), the TF plan dosn't give any error, but the label is not applied.

I receive a similar error, if I try to pass different labels for each subscription (push/pull) of the list.

does anyone have the same problem as me or am I wrong something?

thanks in advance

v3.0.0 not released to registry

There is a v3.0.0 tag but this has not been properly released to the official registry.

Has the tag been created by mistake? If not then it needs releasing otherwise we need to use a workaround like source="[email protected]:terraform-google-modules/terraform-google-pubsub.git?ref=release-v3.0.0"

Unforeseen Behavior on changing filter - bindings missing on first apply run

TL;DR

The error manifests when altering filters. This Was causing the recreation of the pull_subscription object. Consequently, this process leads to the disappearance of access permissions previously assigned to that particular object, impacting the system's functionality.

Expected behavior

The expected behavior is to recreate service bindings, becouse they are missing after the subscription replacement process

Observed behavior

First run (apply):
Replace pull_subscription
Second run (apply)
Creating missing pull_subscription_sa_binding_viewer, pull_subscription_sa_binding_subscriber

Terraform Configuration

module "microservices" {
  source   = "terraform-google-modules/pubsub/google"
  version  = "6.0.0"

  topic              = "microservices-1"
  project_id         = module.project-factory.project_id
  pull_subscriptions = [
    {
      "name" : "microservices-1"
      "ack_deadline_seconds"    = 600
      "max_delivery_attempts"   = 5
      "maximum_backoff"         = "10s"
      "minimum_backoff"         = "5s"
      "enable_message_ordering" = true
      "service_account"   = "[email protected]"
      "expiration_policy" = ""
      "filter"            = "attributes:vendor1"
    }
  ]
}

1. Apply (all resources was created)
2. Change filter
3. Apply (subscription was recreated)
4. Run plan:

- You have now missing bindings to create: `pull_subscription_sa_binding_viewer` and `pull_subscription_sa_binding_subscriber`

Terraform Version

Version 1.6.6

Additional information

Probably needed is to add:

  lifecycle {
    replace_triggered_by = [google_pubsub_subscription.pull_subscriptions]
  }

In my opinion the following addition may introduce other issues that should be carefully reviewed.

Move the publisher service account outside the subscrptions

TL;DR

The roles/pubsub.publisher google_pubsub_topic_iam_member is created for_each subscription, but the permison has no reference to the subscription itselt.

Why the service account that has publish permissions is associated with the subscription? It'd like to see the _iam_member resource outside the subpscription for_each, and maybe receive a list of member to give access

Terraform Resources

The 


resource "google_pubsub_topic_iam_member" "push_topic_binding" {
  for_each = var.create_topic ? { for i in var.push_subscriptions : i.name => i } : {}

  project = var.project_id
  topic   = lookup(each.value, "dead_letter_topic", "projects/${var.project_id}/topics/${var.topic}")
  role    = "roles/pubsub.publisher"
  member  = "serviceAccount:${local.pubsub_svc_account_email}"
  depends_on = [
    google_pubsub_topic.topic,
  ]
}

### Detailed design

```markdown
Instead of `service_account` in the `pull_subscriptions` or `push_subscription`, to have a `publishers` variables on the module.


module "pubsub" {
  source  = "terraform-google-modules/pubsub/google"
  version = "~> 4.0"

  project_id = "my-project"

  create_topic                     = true
  topic                            = "my-topic"

  publishers                       = [
    "[email protected]",
    "[email protected]",
  ]
}

### Additional information

Maybe the service_account on the subscription could be optional

Support Terraform 0.14

The version constraint in this module clamps the Terraform version to < 0.14. Unless there are identifiable incompatibilities, the constraint should be relaxed to allow it to continue functioning now that Terraform 0.14 is GA.

Can't add labels to subscriptions

TL;DR

When adding a map(string) for subscription_labels the following error is returned:

Error: Invalid value for module argument

  on pubsub/topics.tf line 196, in module "main_topic":
 196:   pull_subscriptions  = each.value.subscriptions

The given value is not suitable for child module variable
"pull_subscriptions" defined at
.terraform/modules/pubsub.main_topic/variables.tf:50,1-30: element 0:
element "subscription_labels": string required.

Expected behavior

The label type = subscription was expected to be added to the subscription

Observed behavior

No response

Terraform Configuration

module "main_topic" {
  source  = "terraform-google-modules/pubsub/google"
  version = "~> 5.0"

  project_id = "project-id"
  topic      = "tf-topic"
  topic_labels = {
    type        = "main-topic"
  }
  grant_token_creator = false
  pull_subscriptions  = {
    name                       = "subscription"
    message_retention_duration = "1800s"
    retain_acked_messages      = true
    ack_deadline_seconds       = 600
    enable_message_ordering    = false
    expiration_policy          = ""
    max_delivery_attempts      = 10
    subscription_labels        = {
      type        = "subscription"
    }
  }
}

Terraform Version

1.1.7

Additional information

This probably happens due to pull_subscriptions being declared as a list(map(string)) so it breaks when a map(string) is provided for subscription_labels.