zalando-stups / fullstop Goto Github PK
View Code? Open in Web Editor NEWAudit reporting: collect violations across all AWS accounts
Home Page: https://docs.stups.io/en/latest/components/fullstop.html
License: Apache License 2.0
Audit reporting: collect violations across all AWS accounts
Home Page: https://docs.stups.io/en/latest/components/fullstop.html
License: Apache License 2.0
Find if an instance is in production mode (traffic > 0%)
Passwords are not allowed.
http://docs.aws.amazon.com/IAM/latest/APIReference/API_ListUsers.html
http://docs.aws.amazon.com/IAM/latest/APIReference/API_User.html
PasswordLastUsed != null
Find docker images in pierone that are missing the “deployment.json”
Application/ App-version life cycle. store in a new postgress table
Track ec2 autoscaling events in S3 as well.
Relevant KPI: number of running applications (not instances nor application versions)
Applications "should" be registered in Kio, but this is not (yet) the case and even registered applications could be "inactive" (no running instance anywhere).
One should only get access to a server through 'even'. That implies that you should never use KeyPairs from EC2. Every usage of a KeyPair is wrong and needs a report.
The InstancePlugin throws an exception, when the securitygroup is not found.
Currently violations look like this:
{
"id": 1,
"created": null,
"createdBy": null,
"version": 0,
"lastModified": null,
"lastModifiedBy": null,
"eventId": "87924782487624-da63-535a-83c1-662554adf424",
"accountId": "123456789",
"region": "eu-central-1",
"message": "Instances with ids: [\"i-11111111\"] was started with wrong images: [ami-22222222]",
"violationObject": null,
"comment": null,
"checked": null
}
I suppose it would be helpful to know when a violation was detected by fullstop (also: sorting). Is this timestamp gonna be inside the created
field?
scm-source.json should also be added in fullstop.
http://stups.readthedocs.org/en/latest/user-guide/application-development.html#docker
Make sure that https is always use for communications.
Check that an application only run in a private VPC.
Role generated with the account configurator are not changeable.
Ask to the team for the latest list. ( example: poweruser, admin, read only , fullstop )
Report also usage of unapproves AWS services
ERROR 7326 --- [nio-8080-exec-1] c.a.s.c.p.impl.DefaultExceptionHandler : Invalid UUID string: ESC-359eae033ed748d4a01a
com.amazonaws.services.cloudtrail.processinglibrary.exceptions.ProcessingLibraryException: Invalid UUID string: ESC-359eae033ed748d4a01a
at org.zalando.stups.fullstop.filereader.FileEventReader.readEvents(FileEventReader.java:86)
at org.zalando.stups.fullstop.controller.S3Controller.fetchS3(S3Controller.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:776)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:705)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:618)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:291)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.zalando.stups.fullstop.filter.CORSFilter.doFilter(CORSFilter.java:45)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:102)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:85)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:90)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:516)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1086)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:659)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1558)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1515)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Save all processed information in a S3 bucket
actual: {"accountId":"786011980701","region":"eu-west-1","message":"KeyPair must be blank, but was ["apfeiffer"]"
should also contain the instance id
Find application that are use a wrong AMI
The application should run only in for us valid region.
This should be configurable via env variable
E.g.
GET /violations?accounts=123456789,987654321
Then I could start integrating fullstop into YOUR TURN. Maybe I could also provide the last violation ID known to me, so that I only get the rest.
GET /violations?accounts=123456789,987654321&last_violation=1414
Check that a load balancer can only run in a public Subnet.
Fix credential issue after deployment
Change configuration for log directory in order to simplify changes and deployment.
List of commits:
@mrandi
simplify with value annotations
mrandi authored 26 seconds ago
133f084
@mrandi
add docs in readme
mrandi authored 10 minutes ago
7544c6a
@mrandi
use env variable for logging dir
mrandi authored 13 minutes ago
773456c
@Gregsen
fixed value for download dir
Gregsen authored 18 minutes ago
4085581
@ljaeckel
Merge pull request #65 from zalando-stups/logdirectory …
ljaeckel authored 34 minutes ago
170f5a0
@Gregsen
directory now set via env variable
Gregsen authored 40 minutes ago
7c58ca2
At least hourly report (ideally, asap):
Hints: security groups can be changed during runtime of a server
Add logic for writing violations to a database
Our policy requires access keys to be rotated at least once a week. Check all access keys:
http://docs.aws.amazon.com/IAM/latest/APIReference/API_ListAccessKeys.html
http://docs.aws.amazon.com/IAM/latest/APIReference/API_AccessKeyMetadata.html
CreateDate > now-1week && status = active
I want be able to display ZMON-Checks on a monitor for auditing reasons.
Therefore a ZMON connection from the auditing tool to a data base is needed.
Following checks are necessary:
Find out how to build the report and how to propagate it
Use default Spring-Boot mechanism to define variables. Choose defaults carefully or fail startup if a configuration is not provided.
Add the instance id to the file name to make sure, the file is indeed in the correct folder
Every time we processing again the logs, DB constraint violations throw exceptions.
This should be handled more gracefully.
Check if all application are registered in kio. (https://github.com/zalando-stups/kio)
Find the docker image used for the ec2 instance
Make sure that OAuth is always use.
Store every day the billing information and save it into the S3 bucket
When the keypair plugin logs, it omits the keypair that violates the rules.
Elastic Map Reduce (EMR) instances (using Amazon’s EMR AMIs) MAY be started, but MUST NOT get any access to the central IAM infrastructure through robot users.
All IAM roles of all EMR instances (running Amazon AMIs) must be checked to not allow privilege escalation, i.e. they should not allow downloading security credentials (generated by Mint) from S3.
A reasonable approach would be to only allow white-listed usage of non-Mint S3 buckets (most EMR use cases just need S3 access).
Mint: http://stups.readthedocs.org/en/latest/components/mint.html
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.