redhat-imaging / imagefactory

imagefactory builds images for a variety of operating system/cloud combinations.

#Image Factory#

Image Factory enables appliance creation and deployment to multiple virtualization and Cloud providers.

##Features##

• Build guest images for a growing list of operating system and cloud combinations.
  • Current guest OS support: Fedora 7-19, RHEL 5.x and 6.x
  • Current cloud support: Red Hat Enterprise Virtualization, VMware vSphere, Amazon EC2, Rackspace, OpenStack, and more...
• Image Factory supports "build and upload" or snapshotting of existing images.
• RESTful API makes integrating Image Factory into existing workflows simple.

##Using Image Factory## Building an image begins with a template describing what to build. See an example of such a template below. See the schema documentation for TDL for more detail on creating a template. Note that a template is not tied to a specific cloud.

<template>
    <name>f21</name>
    <os>
        <name>Fedora</name>
        <version>21</version>
        <arch>x86_64</arch>
        <install type='iso'>
            <iso>http://download.fedoraproject.org/pub/fedora/linux/releases/21/Cloud/x86_64/os/</iso>
        </install>
        <rootpw>p@55word!</rootpw>
    </os>
</template>

Ensure to change the element to your desired root password.

Next, use the imagefactory command and specify the template to use and for which clouds to build an image. The above template example was saved to a file name f12_64.tdl.

$ sudo imagefactory --template f12_64.tdl --target ec2

Once the image has been built, use the imagefactory command again, this time to push the image into the cloud.

$ sudo imagefactory --provider ec2-us-west-1 --credentials ec2_credentials.xml

That's it! You can now launch an instance of this image using the cloud provider's management console.

##Installing Image Factory## Installing Image Factory is quick and easy. See the imagefactory rpm installation instructions for more detail.

Dev Setup

If you are wanting to use Imagefactory in a dev environment, then you can run from source. Run the 'imagefactory_dev_setup.sh' script found in the scripts directory. This will setup a dev environment which allows you to run from source. Once this is complete run ./imagefactoryd --foreground to start the server.

Documentation

More documentation on how to configure, use, and develop for imagefactory can be found on the Image Factory website.

imagefactory's Issues

Add support for using remote libvirt instances.

This is primarily useful for the case where users wish to run the Factory within a virtual machine but are able to grant the Factory the ability to create additional VMs on an external infra like RHEV-M or vSphere.

Both Oz and Factory have hard-coded assumptions that virt image files are local and that the libvirt VMs are run under KVM.

Some adventurous users have demonstrated that we can partly work around this by accessing remote RHEV-M image storage via NFS shares. However, in the general case we may have to allow for the case where we build an image remotely then move the image content back to the Factory server for further modification.

This further modification, both by Oz and within the Factory, uses libguestfs. The libguestfs code gracefully deals with a lack of fast local virt (and it works). However it is of course slower than it would be in the presence of KVM. As far as we know, there is no "remote API" mechanism for libguestfs. We need to use it and it needs to run its "mini guest" locally.

An entirely different approach to this issue would be to actively investigate "nested" virtualization. "Nested KVM" (that is, kvm running efficiently within a kvm guest) is in active development. VMWare also claim some level of efficient support for nesting hypervisors.

Confusing activity message on target image completion

When successfully creating a new TargetImage, factory returns a status (via callback) of COMPLETE, but activity is:

activity': 'Initializing image prior to Cloud/OS customization'}

This message is confusing, and made me wonder if the target image was really complete or not. Could this be updated to something more accurate?

Callback URL with username password fail

Factory throws an exception when using HTTP Basic authentication and embedding the credentials in the URL.

e.g. http://admin:password@localhost:3000/base_images

After conversation with sloranz it seems this is not yet supported in ImageFactory. Conductor will use HTTP Auth in the first iteration of Tim integration. sloranz thinks it's not a great deal of effort to add OAuth + HTTP Basic.

Conductor Tim/Factoryv2 integration cut off is 11th Dec. Can get this support a week or so before this date?

We need unit tests for the cli

Add support for running on Ubuntu

Initially this should take the form of PIP or direct from source installs of the stack. Verify that this works first and iron out any bugs that pop up.

When we are confident we run well, look into the process for adding Factory, Oz and whatever additional packages are needed into the official Ubuntu/Debian package tree.

Fix callback registration for image deletion.

This may mean a wholesale change of how we register callbacks for all operations.

Currently the callback is part of the parameters that get passed in as form values. HTTP DELETE doesn't seem to support passing form values, which is a detail that escaped me originally.

Add the ability to validate a combination of provider def and credentials

imagefactory doesn't create directory structure on mounted nfs dir when uploading an image

Imagefactory fails if any of followonf dirs doesn't exist:
/mnt/rhevm-nfs/d6fd140f-796e-478c-916e-c42893f843de
/mnt/rhevm-nfs/d6fd140f-796e-478c-916e-c42893f843de/images
/mnt/rhevm-nfs/d6fd140f-796e-478c-916e-c42893f843de/master
/mnt/rhevm-nfs/d6fd140f-796e-478c-916e-c42893f843de/master/vms

where d6fd140f-796e-478c-916e-c42893f843de is probably cluster id os something like this.
Imagefactory should create these dirs itself.

standardize the status strings

instead of repeating the same status string everywhere, we should be pulling from a constant to avoid typos and to discourage each plugin from defining its own status strings.

Right now there are cases of COMPLETE vs COMPLETED and there is a list of status strings in PersistentImage that do not reflect what is actually being set in the plugins.

add a method to install ImageFactory on a non-RPM distro

Most users in the wild don't use RPMs.

Add fake image id generation for MockSphere

I came across this in testing callbacks with tim, and noticed there is no identifier set for identifier_on_provider. It would be handy to be able to test e2e (well, closer to that) with fake images, so generating some uuid here would be appreciated. For reference, I came across this testing a pull request for tim:
aeolus-incubator/tim#70 (comment)

Create a test infrastructure for secondary factories by using EC2 secondaries

Create a local virtmanager cloud plugin for easy local testing

In an effort to eliminate the barriers to entry and get new users a quick way to see imagefactory in action, we want to add a cloud plugin that would work with oVirt.

Add support for custom sizing and partition schemes in TDL/Factory

Broken image in front page README.md

There's a broken image in the front page README.markdown, try to load from:

https://a248.e.akamai.net/camo.github.com/d9ef3af8d9f8225fec6d5555259eff31cc052b09/687474703a2f2f61656f6c757370726f6a6563742e6769746875622e636f6d2f696d616765666163746f72792f696d616765732f49462d686967686c6576656c2e706e67

Should be fixed. (no idea of the correct url, else this would be a pull request :>)

Make libvirtd failure messages clearer

https://bugzilla.redhat.com/show_bug.cgi?id=858295

For clouds that support it, make the "short description" and "long description" strings configurable.

Most upload style clouds allow you to add a description.

We have generally used our internal UUID as part of the description.

Others have suggesting using the image name given in the TDL.

Make everybody happy and allow this to be configurable.

Argue about what the default should be.

Revise e2eTest.py to optionally use an imagefactoryd REST instance

1. Add arguments for imagefactoryd api url and oauth credentials.
2. When the url is present, hit the REST api instead of using the CLI.

Name reconciliation

https://lists.fedorahosted.org/pipermail/aeolus-devel/2013-January/013579.html

I think the only items that would need to change to support this activity are:
Image Template -> Component Outline
Provider -> Cloud Resource Provider

Delete image should cascade deletion of child images

When deleting a target image, all associated provider images should also be deleted to avoid leaving orphans behind. When deleting a base image, all associated target images and their associated provider images should be deleted.

When deleting, we should delete the children first and only change the status of the parent image (and thus send a callback) when we successfully delete those children and then the parent. If a delete fails for a child, we should set the parent image status to DELETEFAILED and log the actual failure in the status detail so the user knows what image prevented the parent from being deleted.

Rename the plugins

The plugins naming needs some cleanup:

• FedoraOS now supports more than Fedora and should be renamed Oz.
• EC2Cloud -> EC2
• OpenStackCloud -> OpenStack
• MockSphere -> MockCloud
• MockRPMBasedOS -> MockOS

This is a bit of a refactoring effort and means changes to the packages that get released... So it's on the high end of "effort: 2".

Provide a mechanism to get access to generated image files

For Heat we want to generate a collection of images for users to
download and register with their own openstacks. All of the provider
plugins are for uploading to different clouds, but what we really need
is a provider plugin which exports a qcow2 file to the imagefactory
host's local filesystem, with the provider definition specifying the
path etc.

Exposing a URL for downloading the binary image would probably be enough, as long as it is in qcow2 format.

Provide some way to validate a template

There are two options for how this could be done:

1. Generate a RelaxNG document and return it to the client to allow it to perform the validation
2. Add to the plugin API and do the validation at the plugin

I favor option 2.

Add ability to create qcow2 output to Oz.

Document what imagefactory does

We seem to be mistaken as just a REST interface for Oz quite a bit. We need some detailed documentation about:

• what a base image actually is
  • what happens at image creation
  • what happens at image customization
• what imagefactory does to make a target image from a base image
• what imagefactory does to push up a provider image

Support Ubuntu EC2 upload image building

This will require changes to the EC2 Cloud plugin to add key injection to Ubuntu images.

It may be possible to do this by installing cloud-init by default. This has some downsides however. Most notably extensive dependency pull-in.

Add support for building Ubuntu images

Initially, focus on the most recent Ubuntu release. Move backwards as demand dictates.

Oz currently supports Ubuntu JEOS creation but not customization. We will need to add customization. (CORRECTION: Oz supports customization as well.)

The builders, and particularly the EC2 builder, assume a Fedora file layout when doing filesystem modification and adding the key injection code. This will have to be made more flexible.

Other than that it should be simple.

Establish vulnerability reporting procedure

We should probably establish something similar to https://aeolusproject.org/contact.html#security-related

revise e2eTest.py to be compatible with the recent imagefactory CLI changes

We changed the CLI to output logging by default. We need to change the script to use the correct CLI arguments to get the JSON output.

Add entrypoints to root api response

The intent is to make the api more 'discoverable' to the client. I think a side benefit is that you will get a few less questions about what objects exist in the api. Basically, in addition to the (very useful) version information, as a client, I should be able to go the /imagefactory and see all the places I can go from there (/plugins, /base_images, etc).

HTTP 500 when template description contains UTF-8

https://bugzilla.redhat.com/show_bug.cgi?id=860202

Add ability to specify custom install file for base_image

This should be exposed as a valid key in "parameters" for a base image creation.

This should include documentation of the variable substitutions that Oz attempts to do prior to running the install. In particular, ones related to root passwords.

the unsupported platform howto from the front page does not work

Other, unsupported, systems can install Image Factory using setuptools:

$ git clone git://github.com/aeolusproject/imagefactory.git
...
$ cd imagefactory
$ sudo setup.py install

setup.py has no 'x' and has no #!/... line at start

even when I run in using

sudo python setup.py install

it seems ok, but then I get:

/usr/bin/imagefactoryd --debug --no_ssl --no_oauth --foreground
Traceback (most recent call last):
File "/usr/bin/imagefactoryd", line 23, in
from imgfac.PersistentImageManager import PersistentImageManager
File "/usr/lib/python2.7/site-packages/imgfac/PersistentImageManager.py", line 17, in
from ApplicationConfiguration import ApplicationConfiguration
File "/usr/lib/python2.7/site-packages/imgfac/ApplicationConfiguration.py", line 24, in
from imgfac.Version import VERSION as VERSION

which does not seem to be a platform specific issue (FC17)

Make setuptools installation a 1st class citizen

Users should be able to use easy_install or pip instead of having to use RPM.

Change the log message about faulthandler to be more informational

At the moment, the log shows an ImportError if faulthandler is not installed. The text in the log should explain what faulthandler is used for and make it not seem like an error.

We need unit tests for each plugin

Perform basic request validation before responding to a POST - Report failure immediately.

This should be restricted to things that we can do within our running threads. That is, nothing that depends on potentially long running or long timeout external requests should be part of this.

Mainly, we should consider validating the template and other structured data in the request and returning an HTTP error code in response to the initial POST if anything fails. At present, even with a malformed template we can end up with an image object.

Package for Ubuntu 12.04LTS

Related to #157

• Learn how to create a debian package
• Create an imagefactory deb package
• Upload these somewhere public
• Publish documentation on imgfac.org
• Announce availability:imagefactory blog, Twitter, Facebook, Google+

Scope options and tentative effort estimates for OpenStack integration

Support https API endpoints for RHEV-M

Our RHEV-M binding has mandatory CA cert checking for https endpoints.

We either need to find a way to disable that or we need to expand the provider definition syntax to allow passing a CA cert.

Or... we need to check against and application-wide or system-wide set of CA certs.

We need unit tests for the REST API

It's not obvious to me how to write unit tests for the bottle app.

global name 'SubprocessException' is not defined

2012-12-19 10:10:11,825 DEBUG imagefactory_plugins.RHEVM.RHEVM.RHEVM thread(6d478d3b) Message: Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/imagefactory_plugins/RHEVM/RHEVM.py", line 197, in push_image
self.rhevm_push_image_upload(target_image_id, provider, credentials)
File "/usr/lib/python2.7/site-packages/imagefactory_plugins/RHEVM/RHEVM.py", line 218, in rhevm_push_image_upload
ovf_desc = "Template name (%s) from base image (%s)" % (self.tdlobj.name, str(self.builder.base_image.identifier)) )
File "/usr/lib/python2.7/site-packages/imagefactory_plugins/RHEVM/RHEVMHelper.py", line 112, in import_template
self.stage_files()
File "/usr/lib/python2.7/site-packages/imagefactory_plugins/RHEVM/RHEVMHelper.py", line 308, in stage_files
self.mkdir_as_nfs_user(self.ovftmpdir)
File "/usr/lib/python2.7/site-packages/imagefactory_plugins/RHEVM/RHEVMHelper.py", line 155, in mkdir_as_nfs_user
(stdout, stderr, retcode) = subprocess_check_output([ 'mkdir', '%s' % (directory)], preexec_fn=self.become_nfs_user)
File "/usr/lib/python2.7/site-packages/imagefactory_plugins/RHEVM/RHEVMHelper.py", line 59, in subprocess_check_output
raise SubprocessException("'%s' failed(%d): %s" % (cmd, retcode, stderr), retcode)
NameError: global name 'SubprocessException' is not defined

This seems to be a problem with handling error from subprocess. You can reproduce this by removing permissions write permissions to nfs dir for imagefactory.(nfsuser)

BZ845321 - Log details of target_content.xml matching

https://bugzilla.redhat.com/show_bug.cgi?id=845321

Debug log details of what is matched and what isn't when reading target_content.xml

Make specifying a root password optional in the TDL

For Heat images, remote access will be key based so having to hard-code a root password is not desirable.

Support Registration of client apps

This may or may not involve or depend on oauth, but basically my thought here can be described with a flow of steps.

1. Application 1 asks factory to give it an api token of some kind
2. Application 2 does the same
3. Application 1 builds some things, passing in its token to identify it.
4. Application 2, same
5. Application 1 requests a list of base_images that it owns, receives back only those objects.
6. Application, rinse, repeat.

Hopefully that is clear enough, let me know if not.

This, of course, brings up other questions.

• What about users, rather than applications?
• When an app (or user) wants to register, are they automatically granted access, or is some kind of approval system/process needed?
• What is the expectation if an app other than the one granted a given token tries to use that identifier to retrieve a list of objects? If any sensitive data is stored, this could be a big security consideration.
• Does the client need to specify some client id, or is just passing in their api token enough to scope all results?
• Is it possible or desirable to retrieve lists f things created by other clients?

I am sure there are more, this is just a quick list off the top of my head.

vSphere distributed virtual switches may not work - Test

Details are in https://bugzilla.redhat.com/show_bug.cgi?id=853135

Define (and document) the state transitions

We get this question regularly. We should define the workflow (state transitions), provide a mechanism to enforce these (or at least warn when an odd transition happens), and document it in the web documentation for future requests.

change status to BUILDING when the build is started

currently the build status of a base_image sticks to NEW until the build is finished (then moves to COMPLETE); it would be nice to see the status changed to BUILDING after the build process is initiated

Implement a python based CLI which consumes the REST API

Having a client application to consume the REST API would make
scripting much easier. A very nice client could be built with cliff
https://cliff.readthedocs.org/en/latest/ (see python-quantumclient as a
good example)

This would be a lightweight client that can be run from a different host without all of the imagefactory core.

We need unit tests for imgfac