projectjellyfish / jellyfish-fog

General Provisioning of VM's using Fog

Jellyfish Fog

Adds infrastructure, database, and storage product types to Project Jellyfish API, and enables their provisioning on cloud service providers (AWS, Azure, DigitalOcean and VMWare) via the Fog gem.

Installation

Setup Gemfile

Include jellyfish-fog from GitHub in parent Gemfile:

gem 'jellyfish-fog', git: 'git://github.com/projectjellyfish/jellyfish-fog.git

Or include it locally:

  gem 'jellyfish-fog', path: '../jellyfish-fog'

And run bundle install.

Setup Env Vars

Add the following keys which can be set using the .env file or explicitly by the hosted environment (e.g. Heroku):

# GENERAL
JELLYFISH_MOCK_FOG                  = true

# AWS SPECIFIC
JELLYFISH_AWS_ACCESS_KEY_ID           = key
JELLYFISH_AWS_SECRET_ACCESS_KEY       = secret

# AZURE SPECIFIC
JELLYFISH_AZURE_SUB_ID                = azure_subscription_id
JELLYFISH_AZURE_PEM_PATH              = azure_certificate.pem
JELLYFISH_AZURE_API_URL               = https://management.core.windows.net

# VMWARE SPECIFIC
JELLYFISH_VMWARE_USERNAME             = vmware_user
JELLYFISH_VMWARE_PASSWORD             = vmware_password
JELLYFISH_VMWARE_SERVER               = vmware_server
JELLYFISH_VMWARE_EXPECTED_PUBKEY_HASH = vmware_pubkey_hash

Key Descriptions:

• If JELLYFISH_MOCK_FOG is not set to false, Fog will simulate provisioning and retirement rather than actually interacting with the cloud service providers. All keys are required in mock mode, but they can specify dummy data.

• JELLYFISH_AWS_ACCESS_KEY_ID and JELLYFISH_AWS_SECRET_ACCESS_KEY_ID are the public and private keys used by Fog to authenticate to AWS when provisioning and retiring assets.

• JELLYFISH_AZURE_SUB_ID is the Azure subscription id used for provisioning

• JELLYFISH_AZURE_PEM_PATH is location of the client certificate pem file usd to authenticate with Azure. See here for how to generate.

• JELLYFISH_AZURE_API_URL is the API endpoint used for Azure provisioning

• JELLYFISH_VMWARE_USERNAME is the vSphere user

• JELLYFISH_VMWARE_PASSWORD is the vSphere password

• JELLYFISH_VMWARE_SERVER is the vSphere server

• JELLYFISH_VMWARE_EXPECTED_PUBKEY_HASH is the public key hash of the vSphere environment. Can be obtained by passing a dummy value and checking error logs. See here for details.

License

See LICENSE

Copyright 2015 Booz Allen Hamilton

jellyfish-fog's Issues

Azure Support

To Do:

• Modify Jellyfish-fog to bring in Azure Infrastructure Product types
• Port over functionality from Jellyfish-Azure
  • Needs to be modified to auto-select Fog Azure provisioner on checkout
• Port Azure specs
• Update Azure docs with instructions on how to prep VM for image capture

VMWare Support

To Do:

• Modify Jellyfish-fog to bring in VMWare Infrastructure Product type
• Add VMWare provisioner
• Add VMWare specs

Update AWS Provisioner

• Add the following attributes to AWS provisioning questions
  • VPC
  • Subnet
  • Security Group
• Add ability to provision more than t1.micros
  • Solution: Need to specify flavor_id attribute instead instance_id for size of instance
• Add product types for the following
  • VPC
  • Subnet
  • Security Group
• Add ability to add EBS storage (maybe as a new product type?)
• Cut new jellyfish-fog gem
  • Verify AWS provisioning of all product types
  • Push updated gem to RubyGems
  • Update API lockfile with reference to update gem
• Need to decide what kinds of options we want to provide beyond the basic for the newly added product types - to be revisited later

Update environment variables

Update environment variables

To Do:

• Rename environment variables with JELLYFISH_
• Add initializer to update pre-flight-env-check hash

Add AMI ID as product type question for AWS Infrastructure

Add AMI ID as product type question for AWS Infrastructure.

Would enable EC2 instances beyond t1.micro

AWS Provisioning Hangs

If an instance being provisioned is terminated before a response can be returned to its provisioner, the process will hang. Add some kind of timeout.

add support for cloud providers

Actionable Items:

• Add support for AWS
  • Remove Hardcoded AMI for EC2
• Add support for VMware
• Add support for Azure
• Add support for Digital Ocean
• Verify provisioner ~ moved
• Refactor product types into api
• Add specs back into repo that can be run via rake

Follow-up

• soft typing: when a provisioner is enabled then certain key words in a product name will cause it to be routed to the associated provisioner as opposed to NullProvisioner on checkout
• rake jobs:work: this command will hang if the AWS entity being provisioned is killed during pending state and require a SIGKILL to remove.

Refactor provisioners

Refactor provisioners:

• Refactor Payload Response (:default and :raw)
• Refactor Payload Request:
• need to save the payload_request for each product_type
• Refactor Payload acknowledgement (MIQ) - not being used

Error message for AMI ID

It would be good to return an error message if the AMI ID being provisioned comes back as UnAuthorized, otherwise you have no idea that the instance was not actually created. This could be something that should in API.