We've encountered some issues when interacting with a larger number of OAuth2 clients using the provider. Specifically we're seeing the following:
Both issues can be reproduced by creating a sufficient amount of Hydra OAuth2 clients[1] (e.g. 50) against a rate-limited Hydra and then refreshing the state. With the existing provider version, this should cause an error, which causes the panic[2]. After fixing the error.As
issue, we then hit the 429 throttling[2], for which the only immediate recourse without touching the provider is setting Terraform parallelism to something low, which reduces the pressure on the API and avoids getting throttled, but which also then makes certain Terraform runs prohibitively length.
terraform {
required_providers {
hydra = {
source = "svrakitin/hydra"
version = "0.5.2"
}
}
}
provider "hydra" {
endpoint = "https://PROJECT-SLUG.projects.oryapis.com"
authentication {
http_header {
name = "authorization"
value = "bearer ORY_TOKEN"
}
}
}
resource "hydra_oauth2_client" "example" {
count = 50
client_id = "test-client-${count.index}"
client_name = "test-client-${count.index}"
redirect_uris = ["http://localhost:8080/callback"]
response_types = ["code"]
token_endpoint_auth_method = "none"
}
% terraform apply
hydra_oauth2_client.example[19]: Refreshing state... [id=test-client-19]
hydra_oauth2_client.example[17]: Refreshing state... [id=test-client-17]
hydra_oauth2_client.example[25]: Refreshing state... [id=test-client-25]
hydra_oauth2_client.example[3]: Refreshing state... [id=test-client-3]
hydra_oauth2_client.example[42]: Refreshing state... [id=test-client-42]
hydra_oauth2_client.example[8]: Refreshing state... [id=test-client-8]
hydra_oauth2_client.example[1]: Refreshing state... [id=test-client-1]
hydra_oauth2_client.example[29]: Refreshing state... [id=test-client-29]
hydra_oauth2_client.example[5]: Refreshing state... [id=test-client-5]
hydra_oauth2_client.example[40]: Refreshing state... [id=test-client-40]
hydra_oauth2_client.example[35]: Refreshing state... [id=test-client-35]
hydra_oauth2_client.example[23]: Refreshing state... [id=test-client-23]
hydra_oauth2_client.example[2]: Refreshing state... [id=test-client-2]
hydra_oauth2_client.example[27]: Refreshing state... [id=test-client-27]
hydra_oauth2_client.example[22]: Refreshing state... [id=test-client-22]
hydra_oauth2_client.example[13]: Refreshing state... [id=test-client-13]
hydra_oauth2_client.example[28]: Refreshing state... [id=test-client-28]
hydra_oauth2_client.example[46]: Refreshing state... [id=test-client-46]
hydra_oauth2_client.example[31]: Refreshing state... [id=test-client-31]
hydra_oauth2_client.example[0]: Refreshing state... [id=test-client-0]
hydra_oauth2_client.example[4]: Refreshing state... [id=test-client-4]
hydra_oauth2_client.example[14]: Refreshing state... [id=test-client-14]
hydra_oauth2_client.example[48]: Refreshing state... [id=test-client-48]
hydra_oauth2_client.example[26]: Refreshing state... [id=test-client-26]
hydra_oauth2_client.example[21]: Refreshing state... [id=test-client-21]
hydra_oauth2_client.example[38]: Refreshing state... [id=test-client-38]
hydra_oauth2_client.example[49]: Refreshing state... [id=test-client-49]
hydra_oauth2_client.example[39]: Refreshing state... [id=test-client-39]
hydra_oauth2_client.example[11]: Refreshing state... [id=test-client-11]
hydra_oauth2_client.example[33]: Refreshing state... [id=test-client-33]
hydra_oauth2_client.example[47]: Refreshing state... [id=test-client-47]
hydra_oauth2_client.example[18]: Refreshing state... [id=test-client-18]
hydra_oauth2_client.example[32]: Refreshing state... [id=test-client-32]
hydra_oauth2_client.example[12]: Refreshing state... [id=test-client-12]
hydra_oauth2_client.example[7]: Refreshing state... [id=test-client-7]
hydra_oauth2_client.example[37]: Refreshing state... [id=test-client-37]
hydra_oauth2_client.example[44]: Refreshing state... [id=test-client-44]
hydra_oauth2_client.example[15]: Refreshing state... [id=test-client-15]
hydra_oauth2_client.example[20]: Refreshing state... [id=test-client-20]
hydra_oauth2_client.example[45]: Refreshing state... [id=test-client-45]
hydra_oauth2_client.example[41]: Refreshing state... [id=test-client-41]
hydra_oauth2_client.example[10]: Refreshing state... [id=test-client-10]
hydra_oauth2_client.example[34]: Refreshing state... [id=test-client-34]
hydra_oauth2_client.example[36]: Refreshing state... [id=test-client-36]
hydra_oauth2_client.example[24]: Refreshing state... [id=test-client-24]
hydra_oauth2_client.example[9]: Refreshing state... [id=test-client-9]
hydra_oauth2_client.example[43]: Refreshing state... [id=test-client-43]
hydra_oauth2_client.example[6]: Refreshing state... [id=test-client-6]
hydra_oauth2_client.example[30]: Refreshing state... [id=test-client-30]
hydra_oauth2_client.example[16]: Refreshing state... [id=test-client-16]
Planning failed. Terraform encountered an error while generating this plan.
╷
│ Error: Plugin did not respond
│
│ with hydra_oauth2_client.example[6],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│ with hydra_oauth2_client.example[24],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│ with hydra_oauth2_client.example[43],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│ with hydra_oauth2_client.example[16],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│ with hydra_oauth2_client.example[34],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│ with hydra_oauth2_client.example[9],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│ with hydra_oauth2_client.example[30],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│ with hydra_oauth2_client.example[36],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵
Stack trace from the terraform-provider-hydra_v0.5.2 plugin:
panic: errors: target must be a non-nil pointer
goroutine 138 [running]:
errors.As({0x1053c5cd8, 0x1400052f700}, {0x1052f36c0?, 0x0})
errors/wrap.go:103 +0x470
github.com/svrakitin/terraform-provider-hydra/internal/provider.readOAuth2ClientResource({0x1053cccb0, 0x1400062eee0}, 0x14000510c80, {0x105307720?, 0x1400050f260})
github.com/svrakitin/terraform-provider-hydra/internal/provider/resource_oauth2_client.go:343 +0x188
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0x140000e4c40, {0x1053ccc08, 0x14000739620}, 0xd?, {0x105307720, 0x1400050f260})
github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:795 +0xe8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0x140000e4c40, {0x1053ccc08, 0x14000739620}, 0x14000a8cea0, {0x105307720, 0x1400050f260})
github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:1089 +0x430
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0x14000371bc0, {0x1053ccc08?, 0x140007dfef0?}, 0x1400052eb00)
github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/grpc_provider.go:649 +0x3e4
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0x14000255b80, {0x1053ccc08?, 0x140007df710?}, 0x140003b3560)
github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:789 +0x390
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0x105392aa0?, 0x14000255b80}, {0x1053ccc08, 0x140007df710}, 0x14000510980, 0x0)
github.com/hashicorp/[email protected]/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:431 +0x164
google.golang.org/grpc.(*Server).processUnaryRPC(0x1400024a1e0, {0x1053ccc08, 0x140007df5f0}, {0x1053d0db8, 0x1400025d860}, 0x14000566120, 0x14000383980, 0x1058ed9a8, 0x0)
google.golang.org/[email protected]/server.go:1343 +0xb8c
google.golang.org/grpc.(*Server).handleStream(0x1400024a1e0, {0x1053d0db8, 0x1400025d860}, 0x14000566120)
google.golang.org/[email protected]/server.go:1737 +0x990
google.golang.org/grpc.(*Server).serveStreams.func1.1()
google.golang.org/[email protected]/server.go:986 +0x88
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 13
google.golang.org/[email protected]/server.go:997 +0x160
Error: The terraform-provider-hydra_v0.5.2 plugin crashed!
This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.
% terraform apply
╷
│ Warning: Provider development overrides are in effect
│
│ The following provider development overrides are set in the CLI configuration:
│ - svrakitin/hydra in /Users/ttj/projects/metaplay/terraform-provider-hydra
│
│ The behavior may therefore not match any released version of the provider and applying changes may cause the state
│ to become incompatible with published releases.
╵
hydra_oauth2_client.example[27]: Refreshing state... [id=test-client-27]
hydra_oauth2_client.example[39]: Refreshing state... [id=test-client-39]
hydra_oauth2_client.example[6]: Refreshing state... [id=test-client-6]
hydra_oauth2_client.example[38]: Refreshing state... [id=test-client-38]
hydra_oauth2_client.example[19]: Refreshing state... [id=test-client-19]
hydra_oauth2_client.example[17]: Refreshing state... [id=test-client-17]
hydra_oauth2_client.example[32]: Refreshing state... [id=test-client-32]
hydra_oauth2_client.example[46]: Refreshing state... [id=test-client-46]
hydra_oauth2_client.example[22]: Refreshing state... [id=test-client-22]
hydra_oauth2_client.example[48]: Refreshing state... [id=test-client-48]
hydra_oauth2_client.example[33]: Refreshing state... [id=test-client-33]
hydra_oauth2_client.example[34]: Refreshing state... [id=test-client-34]
hydra_oauth2_client.example[18]: Refreshing state... [id=test-client-18]
hydra_oauth2_client.example[12]: Refreshing state... [id=test-client-12]
hydra_oauth2_client.example[45]: Refreshing state... [id=test-client-45]
hydra_oauth2_client.example[49]: Refreshing state... [id=test-client-49]
hydra_oauth2_client.example[20]: Refreshing state... [id=test-client-20]
hydra_oauth2_client.example[37]: Refreshing state... [id=test-client-37]
hydra_oauth2_client.example[5]: Refreshing state... [id=test-client-5]
hydra_oauth2_client.example[9]: Refreshing state... [id=test-client-9]
hydra_oauth2_client.example[29]: Refreshing state... [id=test-client-29]
hydra_oauth2_client.example[0]: Refreshing state... [id=test-client-0]
hydra_oauth2_client.example[47]: Refreshing state... [id=test-client-47]
hydra_oauth2_client.example[42]: Refreshing state... [id=test-client-42]
hydra_oauth2_client.example[25]: Refreshing state... [id=test-client-25]
hydra_oauth2_client.example[2]: Refreshing state... [id=test-client-2]
hydra_oauth2_client.example[4]: Refreshing state... [id=test-client-4]
hydra_oauth2_client.example[13]: Refreshing state... [id=test-client-13]
hydra_oauth2_client.example[23]: Refreshing state... [id=test-client-23]
hydra_oauth2_client.example[40]: Refreshing state... [id=test-client-40]
hydra_oauth2_client.example[8]: Refreshing state... [id=test-client-8]
hydra_oauth2_client.example[43]: Refreshing state... [id=test-client-43]
hydra_oauth2_client.example[31]: Refreshing state... [id=test-client-31]
hydra_oauth2_client.example[10]: Refreshing state... [id=test-client-10]
hydra_oauth2_client.example[36]: Refreshing state... [id=test-client-36]
hydra_oauth2_client.example[44]: Refreshing state... [id=test-client-44]
hydra_oauth2_client.example[28]: Refreshing state... [id=test-client-28]
hydra_oauth2_client.example[11]: Refreshing state... [id=test-client-11]
hydra_oauth2_client.example[26]: Refreshing state... [id=test-client-26]
hydra_oauth2_client.example[7]: Refreshing state... [id=test-client-7]
hydra_oauth2_client.example[14]: Refreshing state... [id=test-client-14]
hydra_oauth2_client.example[16]: Refreshing state... [id=test-client-16]
hydra_oauth2_client.example[15]: Refreshing state... [id=test-client-15]
hydra_oauth2_client.example[24]: Refreshing state... [id=test-client-24]
hydra_oauth2_client.example[1]: Refreshing state... [id=test-client-1]
hydra_oauth2_client.example[30]: Refreshing state... [id=test-client-30]
hydra_oauth2_client.example[21]: Refreshing state... [id=test-client-21]
hydra_oauth2_client.example[41]: Refreshing state... [id=test-client-41]
hydra_oauth2_client.example[35]: Refreshing state... [id=test-client-35]
hydra_oauth2_client.example[3]: Refreshing state... [id=test-client-3]
Planning failed. Terraform encountered an error while generating this plan.
╷
│ Error: 429 Too Many Requests
│
│ with hydra_oauth2_client.example[14],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│ with hydra_oauth2_client.example[16],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│ with hydra_oauth2_client.example[35],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│ with hydra_oauth2_client.example[30],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│ with hydra_oauth2_client.example[41],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│ with hydra_oauth2_client.example[3],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│ with hydra_oauth2_client.example[1],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│ with hydra_oauth2_client.example[24],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│ with hydra_oauth2_client.example[15],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│ with hydra_oauth2_client.example[21],
│ on test.tf line 21, in resource "hydra_oauth2_client" "example":
│ 21: resource "hydra_oauth2_client" "example" {
│
╵