boavizta / cloud-scanner Goto Github PK

Problem

Publish an additional docker version that includes a local copy of Boavizta API (to allow usage with reduced internet access), in which case it could be simpler to build on top of Boavizta API docker image.

Solution

Alternatives

Additional context or elements

Problem

Today, manufacturing impacts of instances are given globally (for the entire lifetime of the underlying infrastructure).
As a user, I would like to get the impacts corresponding to a given period of time.

Solution

Boavizta API v0.2.x brings the possibility to amortize manufacture impacts over a given time (using the Linear Allocation).

• Integrate this amortization functionnality (maybe using optional parameters)
• document it

Alternatives

Additional context or elements

Some aws instance types are not yet in the Boavizta data set.

The current behavior is to query Boavizta API for any instance type, and if the type is not found (resulting in error 500), the error is caught by cloud-scanner, and an empty impact string is returned ({}) for this instance.

It would be more efficent to pre-check which type of instances are supported by Boavizta Dataset and query the API only for relevant instances.

Bug description

When running cloud scanner in docker, the servefunction does not start (nothing seems exposed on 127.0.0.1:8000).

To Reproduce

Expected behavior

JSON OUTPUT

Additional context

This is not only related to the missing expose port in docker file #153 , despite adding it on local test and it does not fix the startup.

Problem

The cloud scanner outputs logs, info messages directly to std err or sdt out.
We would like to be able to configure verbosity from nothing (except errors) to more detailled debug info

Solution

Could use https://docs.rs/loggerv/latest/loggerv/ as a simple solution

See https://deterministic.space/rust-cli-tips.html#bonus-logging

Means updating the CLI options to support multiple verbosity (like -v for info level, - vv for debug level)

Problem

Get the default impacts of an instances/account (i.e. based on instance type an location only).

Solution

Alternatives

Additional context or elements

Original context: in #3

Problem

Support assesment of Instances of Azure Account.

Solution

Feature On hold for the time being because we lack Azure VM impact data in Boavizta Dataset.

Alternatives

Additional context or elements

• Depends on building Azure data set in Boavizta API Boavizta/boaviztapi#66
• #262

A a person who manage an AWS account, I want to run cloud scanner to retrieve the impacts of the resources in my account.

At first, this scanner is intended to be run manually, as a CLI tool, but we should consider wrapping it in a serverless function later.

Problem

Current sls version of cloud-scanner uses hardcoded parameters to do a default scan (defaut, region, defaut api url, 1 hour use time a.s.o)

Solution

Map Inputs parameters of the CLI to the severless app.

Alternatives

Additional context or elements

Problem

Doc is incomplete and Readme is becoming a bit complicated to follow.

Solution

Area to improve

• Building on Linux
• Builing on Windows
• Serverless deployment
• Methodology (i.e. where data come from) with pointer to Boavizta API material
• A general page about the different deployments models (CLI vs serverless, and using private API vs other API)
• A specific chapter about generating metrics (sample rate, data returned)
• Document setup with prometheus and grafana see #114
• A chapter about limitations
• It may better work with a dedicated doc, maybe using mdbook https://github.com/rust-lang/mdBook

Alternatives

Additional context or elements

Problem

Clippy generates a lot of warnings.

Solution

Follow clippy recommendations, update the code to avoid theses warning.

Problem

The generated SDK used to wrap calls to Boavizta API is embeded in the codebase of cloud-scanner.
It makes it heavy and unpractical to maintain (or resuer)

Solution

Extract the Boavizta API Rust SDK to it's own repository and publish it as a crate.
As a first step, this extract will be just a manual copy / publish of existing code (not necessary to automate it).

Alternatives

• (abandonned) Publish the Rust SDK directly from the Boavizta API repository

Additional context or elements

We tried first to publish rhe Rust SDK in the CI chain of Boavizta API repository (see Boavizta/boaviztapi#84), but it turns out it is not very practical because:

• the publish model for crate.io recommends that the code published as crate is available in public repo
• it is generated code (so it is not commited by default inside Boavizta API repo)
• it may require manual adaptations to conform to clippy rules
• it seems easier to maintain in it's own repository (and avoid mixing concerns with Boavizta API repository

Problem

Docker image is unecessary big (~90MB because based on Ubuntu).

Solution

Update dockerfile to use Alpine as a base container.

Alternatives

Additional context or elements

Problem

We can use cloud scanner to export metrics that can be aggregated in a prometheus and displayed in grafana.

Solution

Document this setup, maybe provide a demo dashboard that could easily be imported in grafana.

Problem

At the moment only a limited list of aws_regions is supported.

Solution

We need to match ISO country codes with AWS regions.

Actual region<-> is translation code should be externalized, and even better we should rely on a third party package for this.

Alternatives

Additional context or elements

• Not sure if relevant: https://crates.io/crates/aws-region

Problem

Current version of cloud-scanner only returns defaults impacts of the aws cloud instances (i.e. the usage rate of instances is not considered, only instance types).

Solution

Retrieve instance usage metrics from AWS API and use them when querying Boavizta API to get more realistic results.

Additional context or elements

See meta issue here #3

• More info about how to query metrics statistics (like average cpu utilization for a given time period) of a single instance:https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/US_SingleMetricPerInstance.html

• Python sample code using aws sdk
  https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/example_cloudwatch_GetMetricStatistics_section.html

• And the corresponding function in rust aws sdk:
  https://docs.rs/aws-sdk-cloudwatch/0.11.0/aws_sdk_cloudwatch/client/struct.Client.html#method.get_metric_statistics

• Cloudwatch general concepts:
  https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#dimension-combinations

Example payload format for custom cloud usage query on Boavizta API

{
  "hours_use_time": 2,
  "usage_location": "FRA",
  "workload": {
    "10": {
      "time": 0
    },
    "50": {
      "time": 1
    },
    "100": {
      "time": 0
    },
    "idle": {
      "time": 0
    }
  }
}

Problem

Some part of generated code for embeeded Boaviztat rust SDK generate clippy warning.

Solution

Fix generated code directly in this repo to avoid clippy warnings.

Problem

Docker image is currently build in CI and available only on the private registry of this project.
Cannot be retrieved without authentication / proper permission which make deployment difficult in Enterprise context.

Solution

Publish the image to docker registry.

Not sure if Boavizta org has account / credentials setup for docker registry.

Alternatives

Additional context or elements

Problem

As a new user I would like to quickly test the scanner without compiling or installing anything.

Solution

• Publish a docker image including the precompiled tool.
• Maybe publish also a version that includes a local copy of Boavizta API (to allow usage with reduced internet access), in which case it could be simpler to build on top of Boavizata API docker image.: migrated to issue #21
• Document usage in the readme.

Problem

• gh action
• docker build

Solution

Alternatives

Additional context or elements

Problem

Running the cloud-scanner on a AWS account from outside the account is not the preferred setup for large organizations using AWS, for several reasons:

• potential data leaking outside of the account
  • Inventory / usage data sent to the api (e.g. leaking in api log endpoint)
  • Result data: analysis, inventory (e.g. result exported from aws account)
• requires end user elevated permission (using individual aws IAM user to perform the query)
• need for local setup of the scanner before use.

Solution

Wrap the scanner as a serveless app that can easily be deployed to an aws account. ⚡

This allows:

• to create a custom role for the lambda (and limit the permission to strict minimum)
• audit capability
• easy deployment
• possibility to embed Boavizta API (permitting a kind of offline use to avoid usage data leaking outside the aws account).
• easy to automate scanning

Alternatives

Additional context or elements

Preference for using the serverless framework to easily wrap the rust binary. https://www.serverless.com/

Problem

Currently used plugin (serverless-rust) is left unmaintained

Solution

Test using https://github.com/fdaciuk/sls-rust instead

Alternatives

Additional context or elements

Hello everyone
I hava an issue when I want to execute this command : serverless deploy

My terminal show me an error and after many and many attempts, I don't know what to do.

Here's the error :

Problem

Current version of cloud-scanner only returns defaults impacts of the aws cloud instances (i.e. the usage duration and location of instances is not considered, only instance types).

This means that we retrieve only defaults values.

Solution

Pass the following json body when querying API

{
  "hours_use_time": 2,
  "usage_location": "FRA"
}

Alternatives

Additional context or elements

⚠ We do not consider the workload for the time being, only global use time and location.

Rationale for this is that current workload implementation is under evolution API side, so we wait for implementation of:

• Boavizta/boaviztapi#87
• Boavizta/boaviztapi#86

API doc:

• https://api.boavizta.org/docs#/cloud/instance_cloud_impact_v1_cloud_aws_post

Hello
When I'm trying to build the cloud scanner, there is an issue that stops me. I'm trying to run the following command :
cargo run --bin cloud-scanner-cli standard --hours-use-time 10 | jq

And here is the error :

Do you have a solution please ?
Thanks !

Problem

• Update Readme
• Add a page in doc

Solution

Alternatives

Additional context or elements

Problem

Reduce compile time

Solution

We only deploy using REST API gateway, so we can make use of lambda_http feature flags https://github.com/awslabs/aws-lambda-rust-runtime#feature-flags-in-lambda_http

[dependencies.lambda_http]
version = "0.6"
default-features = false
features = ["apigw_rest"]

Alternatives

Additional context or elements

Problem

We build a demo dashboard to display metrics of cloud scanner but the units and sample rate can be questionned.

Objective of this issue is to discuss which metrics would be relevant to display to an end-user.

Solution

• Maybe display impacts per year of equivalent usage (e.g. this allow easy comparison to a 2 tons objective per example).
• Decide how we attribute the manufacture impacts.

Alternatives

Additional context or elements

Problem

Cargo run asks for a --bin option to specify which binary should start, we could make cli the default.

Solution

Use Cargo default run: The Manifest Format - The Cargo Book

Problem

It would be easier for quick testing if cloud scanner could be embedded with prometheus and grafana in a docker compose file.
But in standalone mode (i.e. non serverless) cloud scanner does not offer a metric http endpoint that can refreshed / scrapped.

Solution

Add an option in CLI to serve metrics continuously on localhost:300/metrics

Alternatives

Additional context or elements

Bug description

Wen using scanner without passing custom API URL, queries fails.

To Reproduce

Run the scanner without passing any custom URL, fails because the default set in code includes a trailing slash (that make requests fail).

Unit tests pass because they use correct URL (only CLI, wich is not tested fails).

Expected behavior

Should use https://api.boavizta.org (without trailing slash) as default.

Problem

Code is a bit hard to follow and there are several interdependencies between modules and data objects.

Solution

Refactor to use more API agnostic (cloud or Boavizta) data objects and "nterfaces" to ease testing.

Alternatives

Additional context or elements

Idea is also to ease the future support of differents cloud providers, impact providers and results exporters.

See https://github.com/Boavizta/cloud-scanner/blob/chore/refactor-for-testing/docs/refactor.mm.md (use a plantuml renderer to see the map).

Problem

Solution

Add link to the github repository and to Boavizta website in the cloud-scanner documentation.

Alternatives

Additional context or elements

Problem

Cloud scanner uses a local version of Boavizta rust SDK that can be outdated compared to API release.

Solution

Migrate to use publicly available SDK (from crate.io) once it i published, see Boavizta/boaviztapi#84

Alternatives

Additional context or elements

Problem

The scan returns all instances of a given account. For large accounts with a lot of instances, we may want to filter results.

Solution

Pass tags to the scan command so that only the instances having theses tags are returned.

Alternatives

Additional context or elements

The general semantic of tags queries on aws seems to be:

• tag1=value1 tag22=value2 : a AND (instance will need to have both tag1 and tag2 to be retained)
• tag1=val1,val2,Val3 : a OR (instance needs to have tag1, with any of the 3 values)

https://gist.github.com/simeji/945165da059d2f4ef4bf

Use a github actions to automate CI.

Should we use cloud query : https://www.cloudquery.io/ ?

Problem

Document minimal AWS role/permissions to use cloud scanner

Solution

Alternatives

Additional context or elements

Problem

Cloud scanner CLI is lacking some options and lacks adaptability.

Solution

CLI covers 2 use cases (subcommands) which needs different flags or set of options.

• get Average (standard) impacts for a given usage duration
• get Mesured impacts: depending on usage rate (use instance workload), in this case we want to use a start and end date

In addition we need to provide

• the region (s) to scan (optional)
• the name for the ouptut file to be saved (optional, default to stdout)
• a list of tags/values that will be used to filter instances (optional)

Alternatives

Additional context or elements

This CLAP tutorial seems to explain all we need : https://blog.logrocket.com/command-line-argument-parsing-rust-using-clap/

Problem

I want to quickly visualize evolution of my scans

Solution

Add routes to returns scan results as OpenMetrics (Prometheus format). This makes easy to plug a grafana visualization.

Alternatives

Additional context or elements

Problem

Documentation is unclear about how to pass aws credentials when use in CLI.
Now only exporting the aws profile through env var (for Linux) is explained... and not in all sections.

See issue #76

Solution

Improve documentation. Describe and test various ways to pass AWS credentials for Unixes and Windows.

Alternatives

Additional context or elements

• https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
• https://docs.aws.amazon.com/powershell/latest/userguide/specifying-your-aws-credentials.html

Problem

By default the scanner only lists resources of the default region of the current AWS profile (or from where the lambda is deployed).

Solution

Allow to scan resources located in a different region.

• The parameter already exists in CLI, it is just not used.
• The region needs to be passed as an optional parameter in the serverless app.

Alternatives

Additional context or elements

Problem

Need a easy way to know the actual version of scanner (when deployed as serverless application).

Solution

Have a basic route in serverless app to return the version of cloud-scanner, or write it to logs.

See https://docs.rs/pkg-version/latest/pkg_version/

Alternatives

Additional context or elements

Problem

I want ot quicky spin up a full demo of cloud-scanner, used to show a real time dashboard.

Solution

A docker compose file to showcase the monitoring use-case:

• cloud-scanner as a metric server (see #149 #148 )
• local boavizta api instance (instead of querying public test api)
• prometheus
• grafana (using demo dashboard)

Alternatives

Additional context or elements

• BoaviztaAPI provides a docker image that should be use to provide a local boavizta API instance (to avoid inventory leak to public API)
• Cloud scanner provides a docker image
• we could use prom/prometheus: https://hub.docker.com/r/prom/prometheus
• grafana/grafana https://hub.docker.com/r/grafana/grafana

🔥 cloud-scanner has to be started with a specific flag to ensure it uses the local API (not the public one)

cloud-scanner -b http://localname/v1 serve

Problem

Get the detailed impacts of instances/account (taking into consideration the usage of the instance through cloud watch metrics or other mechanism).

Solution

Alternatives

Additional context or elements

Original request and way to implement it in #3

Bug description

Both CLI and serverlss version allow using a private (or specific) instance of Boaviztapi as datasource (instead of the default public instance of Boaviztapi).

This is important for enterprise customer setup where we want to be sure that no data of cloud usage leaks outs to a public API.

The parameter exists (as an env var for the serverless version and a CLI parameter) for the CLI version but in practice, both software use the hardcoded URL of the public API.

To Reproduce

Expected behavior

Ensure that

• write code in CLI to use the URL passed as an optional parameter (or default to standard public URL)
• update serverless version makes to use the custom URL passed as environment variable
• both tools provide helpful error message in case the URL is unreacheable
• Give example of how to use this private instance in the Cloud Scanner doc for CLI
• Give example of how to use this private instance in the Cloud Scanner doc for SLS instance

JSON OUTPUT

Additional context

Problem

Support other cloud providers like Azure

Solution

Alternatives

Additional context or elements

May imply CLI option changes See #14

Problem

Errors are not well managed, they can result in program exiting without much information

Solution

Investigate how it could be better managed.

see:

• https://nick.groenen.me/posts/rust-error-handling/
• Using Anyhow to let error bubble up and still add context: https://deterministic.space/rust-cli-tips.html#error-handling

Bug description

Docker file is missing the expose port (8000) requiered to expose a metric server

To Reproduce

Expected behavior

JSON OUTPUT

Additional context