GithubHelp home page GithubHelp logo

zalando-stups / fullstop Goto Github PK

View Code? Open in Web Editor NEW
40.0 17.0 14.0 4.66 MB

Audit reporting: collect violations across all AWS accounts

Home Page: https://docs.stups.io/en/latest/components/fullstop.html

License: Apache License 2.0

Java 98.25% Shell 0.05% Python 0.48% PLpgSQL 1.16% Dockerfile 0.07%

fullstop's Issues

Report number of running applications per accounts

Relevant KPI: number of running applications (not instances nor application versions)

Applications "should" be registered in Kio, but this is not (yet) the case and even registered applications could be "inactive" (no running instance anywhere).

Timestamp of violation

Currently violations look like this:

{
    "id": 1,
    "created": null,
    "createdBy": null,
    "version": 0,
    "lastModified": null,
    "lastModifiedBy": null,
    "eventId": "87924782487624-da63-535a-83c1-662554adf424",
    "accountId": "123456789",
    "region": "eu-central-1",
    "message": "Instances with ids: [\"i-11111111\"] was started with wrong images: [ami-22222222]",
    "violationObject": null,
    "comment": null,
    "checked": null
}

I suppose it would be helpful to know when a violation was detected by fullstop (also: sorting). Is this timestamp gonna be inside the created field?

Always https

Make sure that https is always use for communications.

Some role cannot be changed

Role generated with the account configurator are not changeable.
Ask to the team for the latest list. ( example: poweruser, admin, read only , fullstop )

Report also usage of unapproves AWS services

ProcessingLibraryException: Invalid UUID string

ERROR 7326 --- [nio-8080-exec-1] c.a.s.c.p.impl.DefaultExceptionHandler : Invalid UUID string: ESC-359eae033ed748d4a01a

com.amazonaws.services.cloudtrail.processinglibrary.exceptions.ProcessingLibraryException: Invalid UUID string: ESC-359eae033ed748d4a01a
    at org.zalando.stups.fullstop.filereader.FileEventReader.readEvents(FileEventReader.java:86)
    at org.zalando.stups.fullstop.controller.S3Controller.fetchS3(S3Controller.java:103)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:776)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:705)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:618)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:291)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.zalando.stups.fullstop.filter.CORSFilter.doFilter(CORSFilter.java:45)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:102)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:85)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:90)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:516)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1086)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:659)
    at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1558)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1515)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

Filter violations by account IDs

E.g.

GET /violations?accounts=123456789,987654321

Then I could start integrating fullstop into YOUR TURN. Maybe I could also provide the last violation ID known to me, so that I only get the rest.

GET /violations?accounts=123456789,987654321&last_violation=1414

Change log dir config

Change configuration for log directory in order to simplify changes and deployment.

List of commits:

@mrandi
simplify with value annotations
mrandi authored 26 seconds ago
133f084
@mrandi
add docs in readme
mrandi authored 10 minutes ago
7544c6a
@mrandi
use env variable for logging dir
mrandi authored 13 minutes ago
773456c
@Gregsen
fixed value for download dir
Gregsen authored 18 minutes ago
4085581
@ljaeckel
Merge pull request #65 from zalando-stups/logdirectory …
ljaeckel authored 34 minutes ago
170f5a0
@Gregsen
directory now set via env variable
Gregsen authored 40 minutes ago
7c58ca2

Unprotected services

At least hourly report (ideally, asap):

  • All public IPs with an open port 443 (use https)
  • All servers, that have a public IP and no security group at all.
  • All servers, that have a public IP and ports besides 22 and 443 allowed as input.

Hints: security groups can be changed during runtime of a server

ZMON Connection via DB, priority

I want be able to display ZMON-Checks on a monitor for auditing reasons.
Therefore a ZMON connection from the auditing tool to a data base is needed.

Following checks are necessary:

  1. AWS contains an artifact (live) which doesn't have an application registery entry.
  2. All four approvals for an application version are not done with deployment
  3. Code approver, test approver and deployment approver is the same person
  4. Application version information are not completed
  5. Not allowed repository is in use
  6. No valid repository link
  7. Application information are not completed
  8. The specification link is missing (Commit)
  9. No valid specification link
  10. Commit (Committer) is not done by a member of the delivery team (out of scope für Github.com)
  11. The builder (build creator) is not a member of the delivery team

Take care of db exceptions

Every time we processing again the logs, DB constraint violations throw exceptions.
This should be handled more gracefully.

Billing metric

Store every day the billing information and save it into the S3 bucket

Report EMR instances with inappropriate IAM role(s)

Elastic Map Reduce (EMR) instances (using Amazon’s EMR AMIs) MAY be started, but MUST NOT get any access to the central IAM infrastructure through robot users.

All IAM roles of all EMR instances (running Amazon AMIs) must be checked to not allow privilege escalation, i.e. they should not allow downloading security credentials (generated by Mint) from S3.
A reasonable approach would be to only allow white-listed usage of non-Mint S3 buckets (most EMR use cases just need S3 access).

Mint: http://stups.readthedocs.org/en/latest/components/mint.html

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.