similarweb / finala Goto Github PK

What would you like to be added:
Remove the column of Total Spent Price for all resources
Why is this needed:
This column is confusing and it not needed.

What happened:
The collector does not retrieve the full list of Kinesis streams
What you expected to happen:
All Kinesis Streams should be returned while the collector is running the detection.
How to reproduce it (as minimally and precisely as possible):
Run the collector only for Kinesis streams.

More Info
The following Struct kinesis.ListStreamsInput{} holds

// The name of the stream to start the list with.
ExclusiveStartStreamName *string `min:"1" type:"string"`

Which can be used to determine if the we have used the last table name of not.
See for reference how it is implemented in :
finala/collector/aws/resources/dynamodb.go

What would you like to be added:

1. From UI tweaking the collector configuration would make things easier.
2. Most of the time, we don't know which metric this AWS resource has ? A dynamic drop down display would help us select what metric I need and the value for it.

Why is this needed:

1. UI Will make things easier.

Hello again,

Many thanks for your contribution. I can only access aws account with aws_access_key_id, aws_secret_access_key & aws_session_token together. But I can't see any option to supply aws_session_token in config.yaml. Could you please give your inputs on how this can be achieved.

Best Regards,
Nagaraju

What happened:
The API returns maximum of 100 records per resource although that the resource has more records to return

What you expected to happen:
API should return all the records

How to reproduce it (as minimally and precisely as possible):

What would you like to be added:
Separate resources which are Potential Cost Saving and Unused resources
Why is this needed:
Today we collect the Potential Cost Saving and Unused resources together , but the unused resources don't have a price tag .
We should separate Potential Cost Saving and Unused resources

finala 0.1.2

./finala aws -c finala.yaml 
WARN[0000] setting logging level                         fields.level=
WARN[0000] Invalid log level, not setting                fields.level=
INFO[0000] Setting up storage configuration              connection=DB.db dialect=sqlite3
panic: failed to connect database Binary was compiled with 'CGO_ENABLED=0', go-sqlite3 requires cgo to work. This is a stub

goroutine 1 [running]:
finala/storage.NewStorageManager(0x134fde3, 0x7, 0x134e724, 0x5, 0xc000379170)
        /Users/elad.kaplan/similarweb/git/finala/storage/mysql.go:64 +0x4c0
finala/cmd.initCmd()
        /Users/elad.kaplan/similarweb/git/finala/cmd/root.go:146 +0x187
github.com/spf13/cobra.(*Command).preRun(0x1ee10a0)
        /Users/elad.kaplan/go/pkg/mod/github.com/spf13/[email protected]/command.go:856 +0x49
github.com/spf13/cobra.(*Command).execute(0x1ee10a0, 0xc0003f6f00, 0x2, 0x2, 0x1ee10a0, 0xc0003f6f00)
        /Users/elad.kaplan/go/pkg/mod/github.com/spf13/[email protected]/command.go:792 +0x145
github.com/spf13/cobra.(*Command).ExecuteC(0x1ee1320, 0x1ea2a60, 0xc000050750, 0xc000303f50)
        /Users/elad.kaplan/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/elad.kaplan/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
finala/cmd.Execute()
        /Users/elad.kaplan/similarweb/git/finala/cmd/root.go:68 +0x31
main.main()
        /Users/elad.kaplan/similarweb/git/finala/main.go:8 +0x20

What would you like to be added:

To allow using AWS Profiles from AWS default config files, instead of hardcoding Secret Key in collector.yaml.

Why is this needed:

We have multiple AWS Accounts but the only way to reach to those accounts is to assume the roles from the main account. Since we cannot create new Credentials in all the child accounts, we must create temporary token and session token using STS and use them in collector.yaml configuration file.

The collector.yaml should have an option to use AWS Profiles that assumes roles, instead of hard-coding credentials.

What would you like to be added:
Show in one view multiple execution summary

What would you like to be added:
Looking through the documentation, I didn't see a trivial way to add all AWS accounts in an organization to the providers list.
Maybe the configuration could look something like:

providers:
  aws:
    accounts:
    - name: 'main account'
      org_master: true
      regions:
      - us-east-1
      - us-east-2

Then, Finala could call Organizations.ListAccounts to enumerate these accounts and assume a pre-configured role specified in the AWS configuration file with the appropriate permissions, for example:

[profile main]

[profile member-acct-1]
role_arn = arn:aws:iam::123412341234:role/cost_management
role_session_name = finala
source_profile = main

[profile member-acct-2]
role_arn = arn:aws:iam::987654321012:role/cost_management
role_session_name = finala
source_profile = main

; etc.

Why is this needed:
This would be a very powerful feature to have for organizations that manage a large number of AWS accounts.

I"m using finala helm chart. After making some changes in collector.yaml file and aws.metrics.yaml file. While Upgrading Helm Chart on my local I'm getting this error. Even though I have commented NAT on aws.metrics.yaml. Thanks in advance.

What would you like to be added:
Change the price in the left menu to total spent price per month instead totalSpendPrice

What happened:
When the Finala Docker container is configured to assume a cross-account IAM role, the role is only assumed for reading the resources (i.e. the ReadOnlyAccess policy), but not the pricing:* actions (i.e. the AWSPriceListServiceFullAccess policy).

What you expected to happen:
The assumed role is used for all actions.

How to reproduce it (as minimally and precisely as possible):

1. Have two AWS accounts (I'll call them local and remote, where local has Finala deployed, and remote is the account Finala scans).
2. On the local account create an IAM role allow-assume-role that has just a single permission for the sts:AssumeRole action (with * as Resource to keep it simple).
3. On the remote account create a cross-account IAM role finala-cross-account which trusts the local account, and attach the ReadOnlyAccess and AWSPriceListServiceFullAccess policies to it.
4. Generate temporary credentials using the allow-assume-role role - aws sts assume-role --role-arn "arn:aws:iam::111111111111:role/allow-assume-role" --role-session-name "test-assume".
5. Add the temporary credentials to docker-compose.yaml:

  collector:
      <truncated>
      environment:
        - AWS_ACCESS_KEY_ID=access_key_id_here
        - AWS_SECRET_ACCESS_KEY=secret_key_here
        - AWS_SESSION_TOKEN=session_token_here

6. Add the finala-cross-account role to configuration/collector.yaml:

providers:
  aws:
    accounts:
        <truncated>
        role: arn:aws:iam::222222222222:role/finala-cross-account

7. Start Finala with docker-compose up.
8. You will see in the logs that the collector starts as expected, and no errors are produced when analyzing the resources:

collector_1      | time="2020-12-17T10:49:54Z" level=warning msg="setting logging level" fields.level=info
collector_1      | time="2020-12-17T10:49:54Z" level=info msg="generate collector execution id" id=general_1608202194
collector_1      | time="2020-12-17T10:49:54Z" level=info msg="auth: using aws role" region=
collector_1      | time="2020-12-17T10:49:54Z" level=info msg="auth: using aws role" region=us-east-1
collector_1      | time="2020-12-17T10:49:54Z" level=info msg="auth: using aws role" region=us-east-1
collector_1      | time="2020-12-17T10:49:55Z" level=info msg="starting to analyze resource" region=us-east-1 resource=documentDB
collector_1      | time="2020-12-17T10:49:57Z" level=info msg="starting to analyze resource" region=us-east-1 resource=ec2_volume
collector_1      | time="2020-12-17T10:49:59Z" level=info msg="analyzing resource" region=us-east-1 resource=kinesis
collector_1      | time="2020-12-17T10:50:01Z" level=info msg="Amount of streams" streams_count=0

However, errors similar to the one below can be seen when it tries to perform any pricing action:

collector_1      | time="2020-12-17T10:50:02Z" level=error msg="could not describe pricing product" error="AccessDeniedException: User: <redacted> is not authorized to perform: pricing:GetProducts\n\tstatus code: 400, request id: e3b83592-f6c8-4a71-a287-a72830d399a7" search_query="{\n  Filters: [{\n      Field: \"productFamily\",\n      Type: \"TERM_MATCH\",\n      Value: \"Kinesis Streams\"\n    },{\n      Field: \"group\",\n      Type: \"TERM_MATCH\",\n      Value: \"Provisioned shard hour\"\n    },{\n      Field: \"location\",\n      Type: \"TERM_MATCH\",\n      Value: \"US East (N. Virginia)\"\n    }],\n  ServiceCode: \"AmazonKinesis\"\n}"
collector_1      | time="2020-12-17T10:50:02Z" level=error msg="Could not get shard price" error="AccessDeniedException: User: <redacted> is not authorized to perform: pricing:GetProducts\n\tstatus code: 400, request id: e3b83592-f6c8-4a71-a287-a72830d399a7"

Anything else we need to know?:
The errors can be avoided if the AWSPriceListServiceFullAccess policy is attached to the allow-assume-role role instead of the finala-cross-account role. I first discovered this issue when deploying Finala using the Helm chart into an EKS cluster that runs in one of our AWS accounts, and needs to scan a number of other ("remote") accounts.

Environment:

• Finala version: v0.4.0
• Cloud provider or hardware configuration: AWS EKS in production, Docker 20.10.0 in local reproduction.
• OS (e.g: cat /etc/os-release):

NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.9.6
PRETTY_NAME="Alpine Linux v3.9"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"

• Kernel (e.g. uname -a): Linux 1d47d5c68b49 4.19.121-linuxkit #1 SMP Tue Dec 1 17:50:32 UTC 2020 x86_64 Linux
• Install tools: docker-compose, Helm chart
• Network plugin and version (if this is a network-related bug): N/A
• Others: N/A

What would you like to be added:
We want to have the ability to send notifications from the last executions for all AWS accounts.
Why is this needed:
We want the visibility for all AWS accounts.

What happened:
The UI shows AZ as empty in RDS resources
What you expected to happen:
The UI should present the actual value of the RDS AZ
How to reproduce it (as minimally and precisely as possible):
Look at the UI of Finala , choose the resource RDS

What would you like to be added:

• Multiple teams using same aws accounts needs a way to filter costs by tags.
• Currently you have to select this feature on each page.
• Multiple tags filters should be there so teams can first filter by team tag, then filter by environment tag.

Why is this needed:

1. So cost can be divided by teams by default.
2. Navigate to other pages and team tags is on by default.

What happened:
From time to time the test of collector TestAddEvent fails

What you expected to happen:
The test should always succeed if the code works properly.
How to reproduce it (as minimally and precisely as possible):
You will need to run the test manually until you will success to reproduce Test

What would you like to be added:

• AWS Lambda UI should display total number of innvocation count.

• AWS Lambda UI Should display memory size.

• AWS Lambda collector should check if the function is choosing the right memory size ?

Why is this needed:

• This will help us know the most innvocations used by a lambda function.

• Will help us identify which lambda function used max memory size.

• right memory size predictor will help us choosing the right size for the function.

What would you like to be added:
Add the detection of unused ElasticIPs
Why is this needed:
Additional resource detection.

I have seamlessly built finala on a local vagrant machine and everything seems to work. I also packaged finala into a docker image and I was able to run all the features. However, when I run on a Ubuntu and Amazon Linux box in AWS (either directly on the box or using docker), it seems to run like the vagrant environment, but the dashboard is empty. I have run on 3 different environments (using the same AWS credentials that worked inside vagrant), and all the three environments had empty dashboards.
The config.yaml for all the environments are the same.
Could you point me at where I should be checking?

What happened:
RDS Collector tries to collect events about Neptune which is not part of this resource.

How to reproduce it (as minimally and precisely as possible):
Run the collector for the RDS

What would you like to be added:
The API server does not support filtering resources by Tags we should add it.
Add the option to filter resources by Tags in the API
Why is this needed:
Users can better understand and investigate their specific resources.

What would you like to be added:
There is an API which indicates if the client has the latest version of Finala
Why is this needed:
Clients will be able to know once we release a new version of Finala.

What happened:
RDS calculation is missing the costs of the disk is uses and the backup it stores.

How to reproduce it (as minimally and precisely as possible):
Run the collector for the RDS resources and see the pricing.

What would you like to be added:
Add the option to filters resources by Tags in the UserInterface
Why is this needed:
A consumer of Finala can better investigate unutilized resources according to Tgas.

What would you like to be added:
We want to add tests for our storage
Why is this needed:
We don't have test coverage for our storage

What happened:
The API does not always return the right results for a given ElasticSearch Query.
When our API will get a GET request to the following API:

api/v1/summary/general_1598256010?filter_Data.Tag.owner=usera@magicemail.com

The response will contain other owner which are not [email protected]

What you expected to happen:
We expect to API response to have only the searched query results and nothing else.

How to reproduce it (as minimally and precisely as possible):
Filter by any tag in the UI

Show spot instances price

What would you like to be added:
Add the ability to pass Context and WaitGroups to server utils
Why is this needed:
we want it in order to have one context for all applications running in the API or other Finala component.

What happened:
While the collector detects RDS resource it can not find the right
What you expected to happen:
aurora-mysql instance should return the right price
How to reproduce it (as minimally and precisely as possible):
Run the collector locally
Anything else we need to know?:
│ qual to 1 product" products=0 search_query="{\n Filters: [\n {\n Field: \"databaseEngine\",\n Type: \"TERM_MATCH\",\n Value: \"aurora-mysql\"\n },\n {\n Field: \"instance ││ qual to 1 product" products=0 search_query="{\n Filters: [\n {\n Field: \"databaseEngine\",\n Type: \"TERM_MATCH\",\n Value: \"aurora-mysql\"\n },\n {\n Field: \"instance │

What would you like to be added:
Add Support for Multiple AWS Accounts
Why is this needed:
We need to show multiple AWS accounts via the UI.

The Installation link (https://finala.io/docs/installation/getting-started) in the readme is broken.
https://github.com/similarweb/finala#installation

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Finala version
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

What would you like to be added:
Load dynamic resource detection

Why is this needed:
Today we loading the resources hard-coded collector/aws/run.go. we need to convert the resources to be plugins for dynamic load for better maintenance, implementation and unitesting

What would you like to be added:
Notifier should send a URL to each notification group with the relevant filters
Why is this needed:
We want the notification to have the exact filters for the notification group.

What would you like to be added:
We need to show the data of the Trends API which was recently added
Why is this needed:

What would you like to be added:
Add the option to enable/disable resource detection according to collector's configuration
Why is this needed:
Sometimes we don't want a resource to be detected we want to disable the scanning of this resource from config file.

What would you like to be added:
In Unused resource#
I see you display metric used to evaluate but not the value of the metric.
For example, RDS instances you display the metric used connections . Also display The number of connections it had, even when it had zero connections.

Why is this needed:
It would be nice to know what was metric value of that resource.

What would you like to be added:
Add ElasticSearch resource detection
Why is this needed:
Additional resources to detection.

What happened:

Neptune DB return in RDS list but the price is 0 $

What would you like to be added:
The way most of the companies budget works is according to tags of resources in their AWS Account.
We want to find the resources which are unused and untagged and report them to Finala
Why is this needed:
Each resource that is unsued and untagged might cost a lot of money, we want to be able to identify them as fast as possible.

finala 0.1.2

./finala aws -c finala.yaml --storage-driver "mysql" --storage-connection-string "root@(127.0.0.1)/finala"
WARN[0000] setting logging level                         fields.level=
WARN[0000] Invalid log level, not setting                fields.level=
INFO[0000] Setting up storage configuration              connection="root@(127.0.0.1)/finala" dialect=mysql
panic: failed to connect database sql: unknown driver "mysql" (forgotten import?)

goroutine 1 [running]:
finala/storage.NewStorageManager(0x7ffc490ed0fc, 0x5, 0x7ffc490ed11e, 0x17, 0xc000361170)
        /Users/elad.kaplan/similarweb/git/finala/storage/mysql.go:64 +0x4c0
finala/cmd.initCmd()
        /Users/elad.kaplan/similarweb/git/finala/cmd/root.go:146 +0x187
github.com/spf13/cobra.(*Command).preRun(0x1ee10a0)
        /Users/elad.kaplan/go/pkg/mod/github.com/spf13/[email protected]/command.go:856 +0x49
github.com/spf13/cobra.(*Command).execute(0x1ee10a0, 0xc0002c2d80, 0x6, 0x6, 0x1ee10a0, 0xc0002c2d80)
        /Users/elad.kaplan/go/pkg/mod/github.com/spf13/[email protected]/command.go:792 +0x145
github.com/spf13/cobra.(*Command).ExecuteC(0x1ee1320, 0x1ea2a60, 0xc000052750, 0xc0002eff50)
        /Users/elad.kaplan/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/elad.kaplan/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
finala/cmd.Execute()
        /Users/elad.kaplan/similarweb/git/finala/cmd/root.go:68 +0x31
main.main()
        /Users/elad.kaplan/similarweb/git/finala/main.go:8 +0x20

Why is this needed
Today when we notify a notification group via Finala notifiers we don't print the total amount of potential savings.
It would be great to present the total amount of potential savings per notification group to have better visibility for all the stack.

What would you like to be added:
The UI should support displaying multiple selections of multiple collector executions, this way we will have visibility on all our AWS accounts in one place
Why is this needed:
Multiple AWS accounts visibility in one place

What happened:
while building finala dokcer image I got the following error:

https://github.com/sass/node-sass/releases/download/v4.14.1/linux-arm64-72_binding.node

What you expected to happen:
being able to create finala image
How to reproduce it (as minimally and precisely as possible):
run the docker-compose. / docker build command on macbook with M1 chip
Anything else we need to know?:
the node-sass v4.14 specific build is missing. they didn't generate it
Environment:

• Finala version : latest (from build)
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release): macOs 12.3 Apple M1 Pro
• Kernel (e.g. uname -a): Darwin Kernel Version 21.4.0: Mon Feb 21 20:35:58 PST 2022; root:xnu-8020.101.4~2/RELEASE_ARM64_T6000 arm64

Hi,

I am trying to create build but, I am facing issues. Could look into the issue and give inputs..
Please find log below.

[root@awsws01 finala]# ls
 CHANGELOG.md  LICENSE   README.md      cmd     config.yaml  go.mod  main.go   provider  structs    visibility
Dockerfile     Makefile  build-support  config  expression   go.sum  printers  storage   testutils
[root@awsws01 finala]#  make build
go build -o finala -v
main.go:4:2: cannot find package "finala/cmd" in any of:
	/usr/lib/golang/src/finala/cmd (from $GOROOT)
	/root/go/src/finala/cmd (from $GOPATH)
make: *** [build] Error 1
[root@awsws01 finala]# make --version
GNU Make 3.82
Built for x86_64-koji-linux-gnu
Copyright (C) 2010  Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[root@awsws01 finala]# go version
go version go1.9.4 linux/amd64
[root@awsws01 finala]#

Marker has to be defined in DescribeDBInstancesInput to get more records

Currently, DescribeDBInstances scanning only 100 records. If the AWS account has 150 RDS Instances. Must define Marker value in DescribeDBInstancesInput