As it stands in the UI today map values appear as a single line in the table of values.

This is not an easy to read format for large maps of values. We should try to "pretty print" these values, or parse the maps and present them in the table as separate values. For example, using the screenshot above as a reference, we could have

It's a cool tool and I'd like to know if it is possible to disable S3 versioning scan?
Because our tfstate file has many many version on S3 and I only like to see the latest version on the chart.

Hello,

Is it possible to add ETCD backend support with/without SSL, with/without auth, with/without HA.

https://www.terraform.io/docs/backends/types/etcd.html

I have noticed that Terraboard can't read some of my tfstate files in my bucket. Logs are reporting that xxx.tfstate file has been created with a newer Terraform version that Terraboard can't support yet.
It would be nice just to have some indications when putting a new image on DockerHub, saying what is the latest supported Terraform version. Thanks

Get lock status for statefiles using DynamoDB when available.

First of all, thanks for your amazing work with terrabord!!!!
An absolutely amazing project 💙
But trying to follow Readme instructions with docker I haven't been able to have an up and running environment, I needed to change to docker-compose "way".

background:
Running postgres on an RDS instance
Trying to run terraboard on ECS (EC2 instance)

When I try to run terraboard's docker image on AWS ECS I get an error on startup that the RDS DB instance is busy

level=info msg="Terraboard v0.21.0 (built for Terraform v0.12.28) is starting..."
level=info msg="Using AWS (S3+DynamoDB) as the state/locks provider"
level=fatal msg="dial tcp: lookup terraboard.abcd1234.us-east-1.rds.amazonaws.com: device or resource busy"

I confirmed all my configs/permissions are correct by running the container locally with no issues, also confirmed that the ECS instance the container runs on is able to connect to the RDS instance directly so that end is setup correctly as well.

Running Terraboard in containers, and when I get to a specific state file, the terraboard container will exit this error.

I've tried relaunching the terraboard container and it will fail as it tries to import the same state file. I've also tried removing the db container and starting fresh, and it will fail again on the same state file.

time="2020-04-30T19:52:15Z" level=info msg="Retrieving state from S3" path="path/terraform.tfstate" version_id=id
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xc8db68]

goroutine 34 [running]:
github.com/camptocamp/terraboard/db.marshalAttributeValues(0x0, 0xc0003cf140, 0xc0004e7610, 0x0)
        /go/src/github.com/camptocamp/terraboard/db/db.go:82 +0x78
github.com/camptocamp/terraboard/db.(*Database).stateS3toDB(0xc000368180, 0xc0003cea80, 0xc000027f80, 0x3b, 0xc000358900, 0x20, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/camptocamp/terraboard/db/db.go:70 +0x23a
github.com/camptocamp/terraboard/db.(*Database).InsertState(0xc000368180, 0xc000027f80, 0x3b, 0xc000358900, 0x20, 0xc0003cea80, 0x0, 0x0)
        /go/src/github.com/camptocamp/terraboard/db/db.go:114 +0xa1
main.refreshDB(0x1, 0xc000368180, 0x11f0220, 0xc0001d70e0)
        /go/src/github.com/camptocamp/terraboard/main.go:105 +0xb6b
created by main.main
        /go/src/github.com/camptocamp/terraboard/main.go:164 +0x1c12

It would be nice if an import error occurs, skip the state file and keep importing.

There are many cases where the output of a resource (or an input for that matter) will be marked as sensitive. It would be nice if terraboard was able to recognize this and not display those sensitive values in the UI.

I have opened up hashicorp/terraform#25646 to address the issue of this information not being in the state file.

So I run go get https://github.com/camptocamp/terraboard
and the go install . compiled and installed it.

With following env variables set

AWS_SECRET_ACCESS_KEY=1xxxxxx
AWS_ACCESS_KEY_ID=AKIAJMCDDREEFEEDE
AWS_REGION=ap-southeast-2
AWS_BUCKET=blah-tfstates

λ terraboard                                                             
INFO[0000] Automigrate
INFO[0000] New db is &{<nil> <nil> 0 0xc42024c3c0 false 2 {0xc42022c910} <nil> map[] 0xc42013f830 0x4c25160 0xc42025a560 false}
INFO[0000] Refreshing DB from S3

But running terraboard only create a sqlite db then quit.

What have I done wrong?

Are there plans to create a release that is compatible with Terraform v0.12.26?

I've just tried to use latest version of Docker image (0.19.0) but I just can't connect to postgres:

level=fatal msg="pq: unsupported sslmode "prefer"; only "require" (default), "verify-full", "verify-ca", and "disable" supported"

Same behaviour with 0.18.0.

I've found a "sslmode" version on DockerHub and it's working fine.

Is it a regression? Or, can you point out how to manage sslmode on newer image versions?

are you planing adding support for GCP and Azure?

Add a compare view to compare two versions of a given path.

This means adding an api/compare/<path>?from=<v1>&to=<v2> API point returning a JSON of the processed comparison, and displaying the result in a new compare/<path>?from=<v1>&to=<v2> page.

Would be great to see azure blob support

Hi there!
I was going through the documentation, and I had the feeling that we're getting back to those days, when Microsoft was dictating his rules.
S3, DynamoDB are proprietary and closed source technologies. AWS has never defined a RfC, or tried to create standards or try to reveal their source code and to me it's clear, that if we deal with opensource, we must promote opensource.
Furthermore, AWS is not standard de-facto: I can be on premises, I can have google cloud, and even if, in my case I have AWS (and I really have it) I still don't understand the reasons behind this decision.
Why not using Consul for the backend (which support the session locking)? Why not try something else for the versioning?

Once I upgrade Terraform version to v0.12.0(provider.aws v2.10.0) and "terraform apply" to update state files, then state files disappear in the main dashboard page. Terraborad still can detect that those state files are v0.12.0 but won't display resources within the files.
Could someone suggest, please?

Hi there,

I sucessfully run the terraboard from source, but found that it's only listening on ipv6.
output from netstat:
tcp6 0 0 :::8080 :::* LISTEN 3140/./terraboard

so it doesn't work for http://ipv4:8080.

Could you help me to solve that?

When running a terraform workspaces, "env:/workspace" folders gets created in the s3, however terraboard doesn't pick that up, even when empty AWS_KEY_PREFIX and AWS_FILE_EXTENSION.

The workaround that seems to do it for me was to comment out the if in the GetStates() of s3.go

for _, obj := range result.Contents {
//		if strings.HasSuffix(*obj.Key, ".tfstate") {
			keys = append(keys, *obj.Key)
//		}
	}

and rebuild, then all the possible folders and states files (regardless of extension) show up in the gui - which is the behaviour I needed. Not sure why or how - my go knowledge and time to debug is limited. However since terraboard is an awesome tool (big thanks), thought I shared my finding.

I understand that the dashboard can show attributes of a tfstate file, but is it possible to show outputs as well?
Sometimes we have customized outputs and it will be great if I can see output from the dashboard

I have a bunch of resources created using for_each, however, only 1 resource appears in terraboard.

Seems like the tool is not properly managing the 0.12 syntax features.

This is the same error as #90, but .. I can provide information and not close the ticket. The only thing I masked in the error was our s3 path. Note we're using terraform 12.24.

2020-06-29T12:17:48.896000 {"level":"info","msg":"Retrieving state from S3","path":"our/file","time":"2020-06-29T19:17:48Z","version_id":"0z7Dy7Qpm4SILXmTf5nxO4TJLeuL7RUt"}
2020-06-29T12:17:48.910000 panic: runtime error: invalid memory address or nil pointer dereference
2020-06-29T12:17:48.910000 [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xc8db68]
2020-06-29T12:17:48.910000 goroutine 25 [running]:
2020-06-29T12:17:48.910000 github.com/camptocamp/terraboard/db.marshalAttributeValues(0x0, 0xc00098ab70, 0xc000111610, 0x0)
2020-06-29T12:17:48.910000  /go/src/github.com/camptocamp/terraboard/db/db.go:82 +0x78
2020-06-29T12:17:48.910000 github.com/camptocamp/terraboard/db.(*Database).stateS3toDB(0xc00009f5c8, 0xc00098aa50, 0xc0002acc60, 0x14, 0xc000b2e840, 0x20, 0x0, 0x0, 0x0, 0x0, ...)
2020-06-29T12:17:48.910000  /go/src/github.com/camptocamp/terraboard/db/db.go:70 +0x23a
2020-06-29T12:17:48.910000 github.com/camptocamp/terraboard/db.(*Database).InsertState(0xc00009f5c8, 0xc0002acc60, 0x14, 0xc000b2e840, 0x20, 0xc00098aa50, 0x0, 0x0)
2020-06-29T12:17:48.910000  /go/src/github.com/camptocamp/terraboard/db/db.go:114 +0xa1
2020-06-29T12:17:48.910000 main.refreshDB(0x1, 0xc00009f5c8, 0x11f0220, 0xc0001c1130)
2020-06-29T12:17:48.910000  /go/src/github.com/camptocamp/terraboard/main.go:105 +0xb6b
2020-06-29T12:17:48.910000 created by main.main
2020-06-29T12:17:48.910000  /go/src/github.com/camptocamp/terraboard/main.go:164 +0x1c12

Pulling up that last-logged state file, it is linked to version v0.12.24. Here's the resources and types found in the 1100+ line file:

   1             "type": "AWS",
   1       "type": "aws_api_gateway_account",
   1       "type": "aws_api_gateway_deployment",
   1       "type": "aws_api_gateway_integration",
   1       "type": "aws_api_gateway_integration_response",
   1       "type": "aws_api_gateway_method",
   1       "type": "aws_api_gateway_method_response",
   1       "type": "aws_api_gateway_method_settings",
   1       "type": "aws_api_gateway_resource",
   1       "type": "aws_api_gateway_rest_api",
   1       "type": "aws_caller_identity",
   2       "type": "aws_cloudwatch_log_group",
   1       "type": "aws_iam_account_alias",
   2       "type": "aws_iam_group",
   2       "type": "aws_iam_group_membership",
   2       "type": "aws_iam_group_policy_attachment",
   6       "type": "aws_iam_policy",
   4       "type": "aws_iam_policy_document",
   2       "type": "aws_iam_role",
   5       "type": "aws_iam_role_policy_attachment",
   1       "type": "aws_iam_user",
   1       "type": "aws_kms_alias",
   1       "type": "aws_kms_key",
   1       "type": "aws_lambda_function",
   1       "type": "aws_lambda_permission",
   1       "type": "string"

It's been barfing for a few weeks; we were previously using a dockerhub build sha that supported Terraform 12.20 (and it was failing), but today to fix the panics we updated to terraboard v0.20. It's still failing.

We deleted that one specific state file and things successfully starting up. I'm hoping these clues help to find the cause of the panic. I can provide specific snippets from the state file if you have suspicions about what caused it, but I don't want to have to scrub the entire state file.

Thanks.

There are data references and resource blocks that take up most of the space in state_files. A resource will have "mode": "manged" and a data reference will have "mode": "data" in the state file. This information should be exposed through the UI to give users more context on what they are looking at. For example, right now it is impossible to tell if the aws_vpc that is defined in our one state file is just referencing (via the data object) a pre-made VPC or if the terraform scripts are managing that vpc through a resource declaration. Both appear the same in terraboard.

I am new to Docker and have tried using the docker compose and using the manual commands in GitHub. I tried local and on AWS.

[root@ip-10-30-8-91 terraboard]# docker-compose build && docker-compose up -d
WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
WARNING: The AWS_BUCKET variable is not set. Defaulting to a blank string.
WARNING: The AWS_FILE_EXTENSION variable is not set. Defaulting to a blank string.
WARNING: The AWS_DYNAMODB_TABLE variable is not set. Defaulting to a blank string.
WARNING: The AWS_KEY_PREFIX variable is not set. Defaulting to a blank string.
WARNING: The TERRABOARD_LOG_LEVEL variable is not set. Defaulting to a blank string.
WARNING: The OAUTH_CLIENT_ID variable is not set. Defaulting to a blank string.
WARNING: The OAUTH_CLIENT_SECRET variable is not set. Defaulting to a blank string.
WARNING: The OAUTH_COOKIE_SECRET variable is not set. Defaulting to a blank string.
proxy uses an image, skipping
db uses an image, skipping
Building terraboard
Step 1/12 : FROM golang:1.11 as builder
---> 43a154fee764
Step 2/12 : WORKDIR /go/src/github.com/camptocamp/terraboard
---> Using cache
---> 50f00fdf843d
Step 3/12 : COPY . .
---> Using cache
---> ada28d56f340
Step 4/12 : RUN go get -u github.com/golang/dep/cmd/dep && dep ensure
---> Using cache
---> 06ca804aa2b6
Step 5/12 : RUN make terraboard
---> Using cache
---> 89c25b69d717
Step 6/12 : FROM scratch
--->
Step 7/12 : WORKDIR /
---> Using cache
---> def89a600e59
Step 8/12 : COPY static /static
---> Using cache
---> 207da17b168c
Step 9/12 : COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
---> Using cache
---> 61ce997c2945
Step 10/12 : COPY --from=builder /go/src/github.com/camptocamp/terraboard/terraboard /
---> Using cache
---> da20a6fdeaeb
Step 11/12 : ENTRYPOINT ["/terraboard"]
---> Using cache
---> 89d59becd60b
Step 12/12 : CMD [""]
---> Using cache
---> 078397b52bba
Successfully built 078397b52bba
Successfully tagged camptocamp/terraboard:devel
WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
WARNING: The AWS_BUCKET variable is not set. Defaulting to a blank string.
WARNING: The AWS_FILE_EXTENSION variable is not set. Defaulting to a blank string.
WARNING: The AWS_DYNAMODB_TABLE variable is not set. Defaulting to a blank string.
WARNING: The AWS_KEY_PREFIX variable is not set. Defaulting to a blank string.
WARNING: The TERRABOARD_LOG_LEVEL variable is not set. Defaulting to a blank string.
WARNING: The OAUTH_CLIENT_ID variable is not set. Defaulting to a blank string.
WARNING: The OAUTH_CLIENT_SECRET variable is not set. Defaulting to a blank string.
WARNING: The OAUTH_COOKIE_SECRET variable is not set. Defaulting to a blank string.
Starting terraboard_proxy_1 ...
Starting terraboard_proxy_1 ... done
Starting terraboard_terraboard_1 ... done

Hi.

I have my cli configured with its token id and secret key. It has permission to assume a role in another AWS account, where the s3 bucket and dynamo table are located.

How do I configure terraboard for this setup? Simply passing AWS_SECRET_ACCESS_KEY and
AWS_ACCESS_KEY_ID of the linked account to the terraboard image does not work of course.

The goal here is to have one terraboard showcasing multiplestate files from multiple aws accounts. That would require setting multiple dynamodb tables. Is this possible?

Thanks in advance.

Hi,

First of all thank for developing terraboard, it's a really nice tool!

We are using our own S3 compatible system and currently I think can't change the address with AWS_BUCKET environment variable.

I would like to be able to use AWS_BUCKET=s3://s3.offline.net/bucket_xxx. Maybe I missed something but I think AWS addresses are deeply linked to aws-sdk-go.

Would be nice to have support for multiple s3 buckets for large terraform installations.

Can I use terraboard on local server without AWS infrastructure?

I trying to create local environment like:
Minio (instead S3)
MongoDB (instead DocumentDB)
PostgreSQL (instead RDS PSQL)

API of those services are pretty close to original ones.

1. How can I override S3 endpoint URL and port (for example http://127.0.0.1:1234/bucket_name)
2. How can I set MongoDB URL and port (127.0.0.1:27017)
3. Looks like Terraform Cloud is not necessary requirement if I plan to use only local resources?

Please could you look in to terraboard 0.13.3 support?

It would be very good if terraform outputs also has they own view. I see that the contributor is no longer develop this beautiful tool but maybe some one else can do this.
Thanks a lot.

Hello!

I just attempted an update to 0.22.0 and received a panic on startup:

time="2020-08-14T19:51:13Z" level=info msg="Terraboard v0.22.0 (built for Terraform v0.13.0) is starting..."
panic: terraform.io/builtin/terraform is not a legacy addrs.Provider

Has 0.22.0 dropped support for state file versions older than 0.13.0? This issue describes a similar error which may have been fixed in newer versions of the module? hashicorp/terraform#25803

Hello, this project looks great. Are there any plans to support local state files?

Hi folks,

This is an awesome project!! Thanks for putting it out there.

Some quick feedback on getting the docker image running locally: I had issues getting a link to a postgres container working on Docker for Mac (17.12.0-ce-mac49). It might just be that my VPN setup wasn't playing nice with it, but Docker strongly recommends against link nowadays, and using a shared network is simple and clean to set up.

Instead, I did something along these lines and it worked a treat:

docker network create terranet
docker run --name db \
  -e POSTGRES_USER=gorm \
  -e POSTGRES_DB=gorm \
  -e POSTGRES_PASSWORD="<mypassword>" \
   --net terranet \
  --restart=always postgres -d
docker run -p 8080:8080 \
 -e AWS_REGION="us-east-1" \
 -e AWS_ACCESS_KEY_ID="${ACCESS_KEY_ID}" \
 -e AWS_SECRET_ACCESS_KEY="${ACCESS_KEY}" \
 -e AWS_BUCKET="<bucket>" \
 -e AWS_DYNAMODB_TABLE="<table>" \
 -e DB_PASSWORD="<mypassword>" \
 --net terranet \
 camptocamp/terraboard:latest

On a specific statefile that my bucket contains, I can see the following errors being thrown by postgres:

STATEMENT:  INSERT INTO "attributes" ("resource_id","key","value") VALUES ($1,$2,$3) RETURNING "attributes"."id"
ERROR:  index row size 5048 exceeds maximum 2712 for index "idx_attributes_value"
HINT:  Values larger than 1/3 of a buffer page cannot be indexed.
	Consider a function index of an MD5 hash of the value, or use full text indexing.
STATEMENT:  INSERT INTO "attributes" ("resource_id","key","value") VALUES ($1,$2,$3) RETURNING "attributes"."id"
ERROR:  index row size 4984 exceeds maximum 2712 for index "idx_attributes_value"
HINT:  Values larger than 1/3 of a buffer page cannot be indexed.
	Consider a function index of an MD5 hash of the value, or use full text indexing.
STATEMENT:  INSERT INTO "attributes" ("resource_id","key","value") VALUES ($1,$2,$3) RETURNING "attributes"."id"
ERROR:  index row size 5048 exceeds maximum 2712 for index "idx_attributes_value"

Had a look into the config and the code but couldn't figure out a fix yet.

Use $location.search() in the search page to generate permanent search links.

running Terraboard on my macOS using docker 2.3.0.4
After a while scarping data from terraform.state files I receive the following error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xec9778]

goroutine 28 [running]:
github.com/camptocamp/terraboard/db.marshalAttributeValues(0x0, 0xc0005bc9f0, 0xc00052d5f0, 0x2)
/go/src/github.com/camptocamp/terraboard/db/db.go:83 +0x78
github.com/camptocamp/terraboard/db.(*Database).stateS3toDB(0xc0000c4190, 0xc0005bc600, 0xc0000a2960, 0x51, 0xc00064d1c0, 0x20, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/camptocamp/terraboard/db/db.go:71 +0x25d
github.com/camptocamp/terraboard/db.(*Database).InsertState(0xc0000c4190, 0xc0000a2960, 0x51, 0xc00064d1c0, 0x20, 0xc0005bc600, 0x0, 0x0)
/go/src/github.com/camptocamp/terraboard/db/db.go:115 +0xa1
main.refreshDB(0x1, 0xc0000c4190, 0x156e460, 0xc0001e7d60)
/go/src/github.com/camptocamp/terraboard/main.go:105 +0xb6b
created by main.main
/go/src/github.com/camptocamp/terraboard/main.go:164 +0x1c2f

while running Terraboard container It showing "level=fatal msg="pq: SSL is not enabled on the server""
Please do the needful.

Hi,

I'm sorry to say this - I tried all day long and couldn't figure out how to install the software and make it work.

Tried with go and docker - still, no luck.

Thanks.

Add a filter on path and allow sorting by column on all results.

This means:

• adding a path query parameter to the api/states/stats point, which will filter on queries on '%%'
• adding orderby and order query parameters to the api/states/stats point, which will order by the field (default to last_modified and desc respectively)
• add a filter field on top of the results to enter a path
• have the UI refresh the results when the filter field or sort column are changed

Notes: this should fix #4

I'm not sure how it's meant to be used, but it seems the only way to find the path I want would be to scroll through the 173 pages of entries. I think there should be some form of searching in there (for paths, not resources).

What I'm trying to do

Using ECS Fargate, I'm trying to run Terraboard and the Postgres DB in a single ECS task (two side by side containers). Terraboard fails to start since it cannot find the postgres db.

What is failing

Terraboard attempts to dns lookup 'db' and fails since the postgres db isn't registered in the DNS, its not able to be looked up.

level=fatal msg="dial tcp: lookup db on 172.16.0.2:53: no such host"

Currently with ECS Fargate

Expected behavior

Terraboard should try looking up localhost for the postgres db before fatally ending.

OR

There should be an ability to use hosted postgres (RDS) or a separate container/task.

Fri Sep 15:~/terraboard:~go get -u -v github.com/camptocamp/terraboard

../src/src/github.com/hashicorp/terraform/config/testing.go:9: t.Helper undefined (type *testing.T has no field or method Helper)

I went inside the source and did a 'go build' and could see similar issues

src/github.com/camptocamp/terraboard:~go build .
# github.com/hashicorp/terraform/config
../../../src/github.com/hashicorp/terraform/config/testing.go:9: t.Helper undefined (type *testing.T has no field or method Helper)

Can you please help me with this.

Use $location.search() for paging in overview to generate permanent links.

Fix display of values in sparklines, displaying date & resource count, and allow to click points, linking to the state page on given version.