GithubHelp home page GithubHelp logo

zalando-stups / fullstop Goto Github PK

View Code? Open in Web Editor NEW
40.0 17.0 14.0 4.66 MB

Audit reporting: collect violations across all AWS accounts

Home Page: https://docs.stups.io/en/latest/components/fullstop.html

License: Apache License 2.0

Java 98.25% Shell 0.05% Python 0.48% PLpgSQL 1.16% Dockerfile 0.07%

fullstop's Introduction

Hex.pm Build Status codecov.io swagger-editor Issues in progress Join the chat at https://gitter.im/zalando-stups/fullstop

swagger-validator

Fullstop - Audit reporting

Fullstop AWS overview

Fullstop

Fullstop Architecture overview

Fullstop Architecture

Aim of the project is to enrich CloudTrail log events.

In our scenario we have multiple AWS accounts that need to be handled.

Each of this account has CloudTrail activated and is configured to write in a bucket that resides in the account where also fullstop is running. (Right now in AWS it's not possible to read CloudTrail logs from a different account)

Fullstop will then process events collected from CloudTrail.

To enrich CloudTrail log events with information that comes from other systems than AWS, we should only configure fullstop to do so.

Fullstop can even call the AWS API of a different account, by using a cross-account role. The account that is running fullstop should therefore be trusted by all other accounts in order to perform this operations.

Fullstop-Cross-Account-Role

Plugins

Jobs

In addition to listening on cloudtrail events, Fullstop. runs jobs as well. A list of jobs can be found here.

Application lifecycle

Fullstop. also keeps record of your application lifecycles. for more information, see the [fullstop-application-lifecycle-plugin] (fullstop-plugins/fullstop-application-lifecycle-plugin) and the Application lifecycle API

Information sources

Fullstop uses different source to gather information.

information-sources

Configuration

This environment variables should be set:

FULLSTOP_LOGS
FULLSTOP_SQS_URL
FULLSTOP_SQS_REGION
FULLSTOP_S3_REGION
FULLSTOP_TAUPAGE_OWNERS
FULLSTOP_TAUPAGE_NAME_PREFIX
FULLSTOP_S3_BUCKET
FULLSTOP_KIO_URL
FULLSTOP_PIERONE_URLS
FULLSTOP_TEAM_SERVICE_URL
DATABASE_URL
DATABASE_USER
DATABASE_PASSWORD
DATABASE_DRIVER
INSTANCE_LOGS_S3_BUCKET
ACCESS_TOKEN_URI
CREDENTIALS_DIR
TOKEN_INFO_URI
FULLSTOP_UNAPPROVED_SERVICES_AND_ROLE_BUCKET_NAME
FULLSTOP_UNAPPROVED_SERVICES_AND_ROLE_PREFIX

Example:

$ export FULLSTOP_LOGS=/fullstop_logs_dir
$ export FULLSTOP_SQS_URL=https://sqs.eu-central-1.amazonaws.com/ACCOUNT_ID/fullstop
$ export FULLSTOP_SQS_REGION=eu-central-1
$ export FULLSTOP_S3_REGION=eu-west-1
$ export FULLSTOP_TAUPAGE_OWNERS=999999999999
$ export FULLSTOP_TAUPAGE_NAME_PREFIX=Taupage
$ export FULLSTOP_S3_BUCKET=fullstop-bucket
$ export FULLSTOP_KIO_URL: https://application.registry.address
$ export FULLSTOP_PIERONE_URLS: https://docker.repository.address
$ export FULLSTOP_TEAM_SERVICE_URL: https://team.service.address
$ export DATABASE_URL='jdbc:postgresql://localhost:5432/fullstop'
$ export DATABASE_USER=postgres
$ export DATABASE_PASSWORD='{cipher}234laksnfdlF83NHALF'
$ export DATABASE_DRIVER=org.postgresql.Driver
$ export INSTANCE_LOGS_S3_BUCKET=my-s3-bucket
$ export ACCESS_TOKEN_URI=accessTokenUri
$ export CREDENTIALS_DIR=/location/credentials
$ export TOKEN_INFO_URI=tokenInfoUri
$ export FULLSTOP_UNAPPROVED_SERVICES_AND_ROLE_BUCKET_NAME=fullstop-bucket-policy
$ export FULLSTOP_UNAPPROVED_SERVICES_AND_ROLE_PREFIX=folder_containing_templates_files

Disable CloudTrail Processing

Set the parameter fullstop.container.autoStart=false either as program argument, or as system property to start Fullstop without CloudTrail processing.

Scopes

In order to create violations and save them to the database, your application which is responsible for that, needs this oAuth2 scope for the POST method on "/api/violations" endpoint

fullstop.violation.write

Database setup

Fullstop will store the violations in a RDBMS. Once you start Fullstop, it will create the necessary schema and tables for you. The database itself, however, has to be created by you. Your database password is encrypted with AWS KMS. We are using Taupage to decrypt the password on the fly. To use Amazons KMS for de/encryption, you need to to provide a region and the key id for your key. In Fullstop, both will be provided via environment variables.

The password should be already encrypted, when you store it in the DATABASE_PASSWORD environment variable. An encrypted password always starts with aws:kms:. You can use our CLI tool for encryption or you use Amazons AWS CLI.

Propose API changes

Important all changes should be swagger 2.0 spec compliant.

How to build

Needs Java 1.8. Newer jdks are not supported yet.

$ ./mvnw clean install

Run entire test suite (requires a local docker environment)

$ ./mvnw clean verify -Pintegration-test

How to run

You need a locally running PostgreSQL 9.4 server (database "fullstop", host "localhost", user "postgres", password "postgres").

$ # set env variables first
$ cd fullstop
$ mvn spring-boot:run -Dfullstop.container.autoStart=false

How to build a docker image

Build fullstop:

$ mvn clean package -U

Build scm-source.json:

$ ./scm-source.sh

Build docker image:

$ docker build -t registry/fullstop:0.1 fullstop

Show images:

$ docker images

Run with docker:

$ docker run -it registry/fullstop:0.1

Push docker image:

$ docker push registry/fullstop:0.1

License

Copyright © 2015 Zalando SE

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

fullstop's People

Contributors

aermakov-zalando avatar christianlohmann avatar csenol avatar dependabot-support avatar gregsen avatar hjacobs avatar jbellmann avatar ljaeckel avatar lukasniemeier-zalando avatar mawenzel avatar mrandi avatar prayerslayer avatar roskenet avatar sebastianpoeplau avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fullstop's Issues

Report number of running applications per accounts

Relevant KPI: number of running applications (not instances nor application versions)

Applications "should" be registered in Kio, but this is not (yet) the case and even registered applications could be "inactive" (no running instance anywhere).

Filter violations by account IDs

E.g.

GET /violations?accounts=123456789,987654321

Then I could start integrating fullstop into YOUR TURN. Maybe I could also provide the last violation ID known to me, so that I only get the rest.

GET /violations?accounts=123456789,987654321&last_violation=1414

ProcessingLibraryException: Invalid UUID string

ERROR 7326 --- [nio-8080-exec-1] c.a.s.c.p.impl.DefaultExceptionHandler : Invalid UUID string: ESC-359eae033ed748d4a01a

com.amazonaws.services.cloudtrail.processinglibrary.exceptions.ProcessingLibraryException: Invalid UUID string: ESC-359eae033ed748d4a01a
    at org.zalando.stups.fullstop.filereader.FileEventReader.readEvents(FileEventReader.java:86)
    at org.zalando.stups.fullstop.controller.S3Controller.fetchS3(S3Controller.java:103)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:776)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:705)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:618)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:291)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.zalando.stups.fullstop.filter.CORSFilter.doFilter(CORSFilter.java:45)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:102)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:85)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:90)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:516)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1086)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:659)
    at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1558)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1515)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

Billing metric

Store every day the billing information and save it into the S3 bucket

ZMON Connection via DB, priority

I want be able to display ZMON-Checks on a monitor for auditing reasons.
Therefore a ZMON connection from the auditing tool to a data base is needed.

Following checks are necessary:

  1. AWS contains an artifact (live) which doesn't have an application registery entry.
  2. All four approvals for an application version are not done with deployment
  3. Code approver, test approver and deployment approver is the same person
  4. Application version information are not completed
  5. Not allowed repository is in use
  6. No valid repository link
  7. Application information are not completed
  8. The specification link is missing (Commit)
  9. No valid specification link
  10. Commit (Committer) is not done by a member of the delivery team (out of scope für Github.com)
  11. The builder (build creator) is not a member of the delivery team

Take care of db exceptions

Every time we processing again the logs, DB constraint violations throw exceptions.
This should be handled more gracefully.

Unprotected services

At least hourly report (ideally, asap):

  • All public IPs with an open port 443 (use https)
  • All servers, that have a public IP and no security group at all.
  • All servers, that have a public IP and ports besides 22 and 443 allowed as input.

Hints: security groups can be changed during runtime of a server

Timestamp of violation

Currently violations look like this:

{
    "id": 1,
    "created": null,
    "createdBy": null,
    "version": 0,
    "lastModified": null,
    "lastModifiedBy": null,
    "eventId": "87924782487624-da63-535a-83c1-662554adf424",
    "accountId": "123456789",
    "region": "eu-central-1",
    "message": "Instances with ids: [\"i-11111111\"] was started with wrong images: [ami-22222222]",
    "violationObject": null,
    "comment": null,
    "checked": null
}

I suppose it would be helpful to know when a violation was detected by fullstop (also: sorting). Is this timestamp gonna be inside the created field?

Some role cannot be changed

Role generated with the account configurator are not changeable.
Ask to the team for the latest list. ( example: poweruser, admin, read only , fullstop )

Report also usage of unapproves AWS services

Change log dir config

Change configuration for log directory in order to simplify changes and deployment.

List of commits:

@mrandi
simplify with value annotations
mrandi authored 26 seconds ago
133f084
@mrandi
add docs in readme
mrandi authored 10 minutes ago
7544c6a
@mrandi
use env variable for logging dir
mrandi authored 13 minutes ago
773456c
@Gregsen
fixed value for download dir
Gregsen authored 18 minutes ago
4085581
@ljaeckel
Merge pull request #65 from zalando-stups/logdirectory …
ljaeckel authored 34 minutes ago
170f5a0
@Gregsen
directory now set via env variable
Gregsen authored 40 minutes ago
7c58ca2

Always https

Make sure that https is always use for communications.

Report EMR instances with inappropriate IAM role(s)

Elastic Map Reduce (EMR) instances (using Amazon’s EMR AMIs) MAY be started, but MUST NOT get any access to the central IAM infrastructure through robot users.

All IAM roles of all EMR instances (running Amazon AMIs) must be checked to not allow privilege escalation, i.e. they should not allow downloading security credentials (generated by Mint) from S3.
A reasonable approach would be to only allow white-listed usage of non-Mint S3 buckets (most EMR use cases just need S3 access).

Mint: http://stups.readthedocs.org/en/latest/components/mint.html

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.