target / grease

Automated Scripting Engine For the Modern Age

GREASE

Our Trello Board for Feature Tracking

Automation as a Service

GREASE

• Guest
• Reliability
• Engineering
• Automation
• Solution
• Engine

What it does

GREASE is designed as a system to enable generalized and large scale automation efforts. Rather than approaching automation via manual scripts, scheduled runs in a task runner system, GREASE is designed to be completely hands free. This enables true and actual automation and reduction of toil. GREASE is designed to stop the decades old mentality of operations and operators from break/fix work, and redirect their efforts to the principles of SRE. A common simile for GREASE is: "Let Operations STOP fighting all these fires (automated preventable incidents), install the sprinkler system (GREASE) and begin performing arson investigation (helping solutions portfolio working on & prioritizing their backlog/Designing better processes and infrastructure)

GREASE provides a simple JSON configuration schema for issue detection and a safe Python command class for implementing resolutions.

How it works

GREASE runs 24/7/365 monitoring sources you define and based on configuration can act on these sources. These actions can be anything you can do in python.

Out of the box GREASE is very minimal, it is but an engine. Similar to Django, you write your application, GREASE just serves it.

Getting the Docs

1. Install sphinx pip install sphinx
2. generate the docs by running make html
3. Use a web browser to read the docs starting at <project root>/docs/_build/index.html

Requirements

1. Python2.7
2. Pip
   • requests
   • pymongo
   • psutil
   • psycopg2
   • elasticsearch
   • psutil
   • FOR WINDOWS ONLY pypiwin32

Installing

Via PIP

Simply run pip install tgt_grease

Manually

1. Clone this repo to your machine
2. from the created directory run python setup.py install

grease's Issues

Notifier needs to use Config

GREASE Issue Template Update Notifier to use Configuration

USE THIS TEMPLATE FOR YOUR ISSUE!

• Primary Contact: Grant Gordon
• Team: Bad Grammar and the Solo Queue

Abstract

Notifier still uses self._hipchat* for config values rather than using the config object

Supporting Elements

Issue Checklist

• Maintainer has viewed
• Loaded into Backlog
• PR Submitted [Link Here](Eventual Link)

GREASE Registration Strength

GREASE: Registration Bug

USE THIS TEMPLATE FOR YOUR ISSUE!

• Primary Contact: Grant Gordon
• Team: Target

Abstract

Registration of nodes to the database should happen before anything else but we are still very exposed to potential SegFault since in the cases listed below we could be "unregistered". We should implement more robust and centralized registration checking/error checking

Supporting Elements

• GREASE skips registration when an ID file is already there, without verifying with the DB:
  https://github.com/target/grease/blob/master/tgt_grease_enterprise/LaunchCtl.py#L66-L72
• GREASE attempts to report Telemetry, without making sure that self._grease_identity is in the database.
  https://github.com/target/grease/blob/master/tgt_grease_core_util/Telemetry.py#L35-L51
• I could tear down and rebuild the Dockerized GREASE database, but the server ID file on my local computer remained. So, when I went to register again, GREASE skipped the registration because I already had an ID file, and bailed when trying to report metrics.

Issue Checklist

• Maintainer has viewed
• Loaded into Backlog
• PR Submitted [Link Here](Eventual Link)

Extending from AutomationTest causes local testing to fail

Extending from AutomationTest causes local testing to fail

USE THIS TEMPLATE FOR YOUR ISSUE!

• Primary Contact: Cory Hollinger

Abstract

When implementing a unit test class that extends from AutomationTest, GREASE tries to grab an instance of Mongo when AutomationTest.ioc is initialized. It does this because Mongo configurations are a valid option, but not all configurations will need Mongo and not all developers will have (or need) a Mongo instance stood up.

What needs to be done:

• Add a check to see if Mongo exists on the machine
• If Mongo isn't present, suppress the exception and don't try to grab configs from Mongo

Supporting Elements

File "/usr/local/lib/python3.6/site-packages/tgt_grease/core/Types/test.py", line 66, in init
self.detect = Detect(self.ioc)
File "/usr/local/lib/python3.6/site-packages/tgt_grease/enterprise/Model/Detection.py", line 30, in init
self.ioc.ensureRegistration()
File "/usr/local/lib/python3.6/site-packages/tgt_grease/core/InversionOfControl.py", line 99, in ensureRegistration
'activationTime': datetime.utcnow()
File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 683, in insert_one
session=session),
File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 599, in _insert
bypass_doc_val, session)
File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 579, in _insert_one
True, _insert_command, session)
PYMONGO STACK TRACE OMITTED FOR BREVITY
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 61] Connection refused

Issue Checklist

• Maintainer has viewed
• Loaded into Backlog
• PR Submitted [Link Here](Eventual Link)

Server Not Responding to Windows SysCalls

Server Not Responding to Windows SysCalls

USE THIS TEMPLATE FOR YOUR ISSUE!

• Primary Contact: James Bell
• Team: Target

Abstract

The Windows Service doesn't detect the kill signal from windows services via sc or the UI. This will require a "hard kill" in-order to cycle the server

Supporting Elements

This line from earlier and this one need to be ported about to here and here respectively to restore this functionality

Issue Checklist

• Maintainer has viewed
• Loaded into Backlog
• PR Submitted #62

Typo Causes kill-server Failures

Typo Causes kill-server Failures

USE THIS TEMPLATE FOR YOUR ISSUE!

• Primary Contact: Grant
• Team: Independent

Abstract

A typo in a SQL statement causes kill-server to fail from the command line.

Supporting Elements

grease/tgt_grease_enterprise/Article14Section31.py

Line 157 in 7a387b6

stmt = update(ServerHealth).where(ServerHealth.server == server_id).values(docter=self._config.node_db_id())

https://gist.github.com/gamagori/99490bec609b3c053539a22b015087be

Issue Checklist

• Maintainer has viewed
• Loaded into Backlog
• PR Submitted [Link Here](Eventual Link)

Batch command type

Batch command type

USE THIS TEMPLATE FOR YOUR ISSUE!

• Primary Contact: Cory Hollinger

Abstract

There could be a use case for a hybrid command type with parts of normal Commands and ScheduledCommands.

These commands would go through the normal sourcing -> detecting -> scheduling flow, but would have a "timeToRun" a la scheduled commands. It would collect sourced data in its variable_storage, then act on them in bulk when timeToRun is true.

Issue Checklist

• Maintainer has viewed
• Loaded into Backlog
• PR Submitted [Link Here](Eventual Link)

Central Scheduling Should Not Hard Require the IoC

Central Scheduling Should Not Hard Require the IoC

USE THIS TEMPLATE FOR YOUR ISSUE!

• Primary Contact: James E. Bell Jr.
• Team: Target

Abstract

The Central Scheduling Object should default to none for the __init__ to by default encapsulate it's dependencies but be friendly to Dependency Injection

Supporting Elements

Code Link

From this:

def __init__(self, ioc):
        if isinstance(ioc, GreaseContainer):
            self.ioc = ioc
        else:
            self.ioc = GreaseContainer()
        self.ioc.ensureRegistration()

To this:

def __init__(self, ioc=None):
        if isinstance(ioc, GreaseContainer):
            self.ioc = ioc
        else:
            self.ioc = GreaseContainer()
        self.ioc.ensureRegistration()

Issue Checklist

• Maintainer has viewed
• Loaded into Backlog
• PR Submitted https://github.com/target/grease/pull/59

Optimize Base Class Utilization for Detectors

More Use of Abstract Class by default for Detectors

If you aren't from Target make sure to reach out to James, click here to email, to get a contributor agreement signed!

USE THIS TEMPLATE FOR YOUR ISSUE!

• Primary Contact: Your Name Here
• Team: Your Team Name (If Not Target)

Abstract

We can utilize the base class for detectors and take out the need to define get_result by default. We just need to set a default property and return it in the default implementation of the class

Supporting Elements

Take this line here <- (direct link to code)

and transform it to this:

        _source_data = {}

Then transform this method <- (direct link to code) from:

 @abstractmethod
    def get_result(self):
        # type: () -> dict
        pass

to this:

    def get_result(self):
        # type: () -> dict
        if isinstance(self._source_data, dict):
            return self._source_data
        else:
            raise TypeError("Invalid property type for _source_data::expected dict got [{0}]".format(str(type(self._source_data))))

Issue Checklist

• Maintainer has viewed
• Loaded into Backlog
• PR Submitted [Link Here](Eventual Link)

Change to utilize UTC time in Router

Change datetime.now() to datetime.utcnow()

If you aren't from Target make sure to reach out to James, click here to email, to get a contributor agreement signed!

USE THIS TEMPLATE FOR YOUR ISSUE!

• Primary Contact: Your Name Here
• Team: Your Team Name

Abstract

Change the places where we use local time to UTC time in the router

Supporting Elements

Change This Line <- Link to exact line

 _current_real_second = datetime.now().second

to

 _current_real_second = datetime.utcnow().second

Change This Line <- Link to exact line

 _current_run_second = datetime.now().second

to

 _current_run_second = datetime.utcnow().second

Change This Line <- Link to exact line

stmt = update(JobQueue).where(JobQueue.id == job_id).values(in_progress=False, completed=True, complete_time=datetime.now()

to

stmt = update(JobQueue).where(JobQueue.id == job_id).values(in_progress=False, completed=True, complete_time=datetime.utcnow()

Change This Line <- Link to exact line

return datetime.now().second

to

return datetime.utcnow().second

Issue Checklist

• Maintainer has viewed
• Loaded into Backlog
• PR Submitted [Link Here](Eventual Link)

URL Source

URL Source

USE THIS TEMPLATE FOR YOUR ISSUE!

• Primary Contact: Grant Gordon
• Team: Independent

Abstract

Create GREASE's first open-source Source to serve as an example for people who want to create their own source. This Source will allow GREASE to consume hosts and API endpoints as data sources.

Supporting Elements

Capabilities:

• Ping hostnames
• Ping IP address, by array or by subnet
• Other TBD, pending conversation with James tomorrow.

Assorted To-Do:

• Move BaseSourceClass to the Sources folder, like the BaseDetectorClass
• Provide a sample config. Maybe ensure localhost is online?
• Add logging to DetectorConfiguration.py to ensure people without configs to reference can understand what's going on.

Issue Checklist

• Maintainer has viewed
• Loaded into Backlog
• PR Submitted [Link Here](Eventual Link)

Bool to String Causing Telemetry Issues

Bool to String Causing Telemetry Issues

USE THIS TEMPLATE FOR YOUR ISSUE!

• Primary Contact: Grant Gordon

Abstract

A str(boolean_value) passed to SQLAlchemy throws errors, since SQLAlchemy is expecting a bool, but we're passing a string.

Supporting Elements

See gist here. You can see on that we cast the success of the command we're going to report to a string. SQLAlchemy is expecting a boolean, and this caused issues for me, running SQLAlchemy (1.2.0b3).

Issue Checklist

• Maintainer has viewed
• Loaded into Backlog
• PR Submitted [Link Here](Eventual Link)

Command Exceptions log based on the base file rather than the command

USE THIS TEMPLATE FOR YOUR ISSUE!

• Primary Contact: James Bell
• Team: Target

Abstract

The command exception always shows Command.py and line 106 since that is the trace for the exception since #72 has been merged

Supporting Elements

See Abstract

Issue Checklist

• Maintainer has viewed
• Loaded into Backlog
• PR Submitted [Link Here](Eventual Link)

AutomationTest Support Multiple Mocked Sources

AutomationTest Support Multiple Mocked Sources

USE THIS TEMPLATE FOR YOUR ISSUE!

• Primary Contact: Ronald Queensen
• Team: Target/Core

Abstract

When wanting to mock multiple sources (EX: Multiple URL's in a URL prototype config) the tests fail

Supporting Elements

Given This Config

{
  "name": "demo_1",
  "source": "url_source",
  "job": "demo",
  "exe_env": "general",
  "url": ["http://localhost:8000", "google.com"],
  "logic": {
      "Regex": [
          {
              "field": "url",
              "pattern": ".*",
              "variable": true,
              "variable_name": "url"
          }
      ],
      "Range": [
          {
              "field": "status_code",
              "min": 199,
              "max": 201
          }
      ]
  }
}

And this Test

 from tgt_grease.core.Types import AutomationTest
 from tgt_grease.core import Configuration
 from example_package.Commands import demo
 
 
 class TestDemo1(AutomationTest):
 
     def __init__(self, *args, **kwargs):
         AutomationTest.__init__(self, *args, **kwargs)
         self.configuration = "fs://{0}".format(Configuration.greaseDir + 'etc/basic.config.json')
         self.mock_data = { 
             'url': 'localhost:8000',
             'status_code': 200 
         }   
         self.expected_data = { 
             'url': ['localhost:8000', ''] 
         }   
         self.enabled = True
 
     def test_command(self):
         d = demo()
         self.assertTrue(d.execute({}))

The test fails

Issue Checklist

• Maintainer has viewed
• Loaded into Backlog
• PR Submitted [Link Here](Eventual Link)