fedora-infra / fedimg

Service to automatically upload built Fedora images to internal and external cloud providers.

Fedimg

Service to automatically upload built Fedora images to internal and external cloud providers.

License

Fedimg is licensed under AGPLv3 or newer. See the LICENSE file for more information.

Documentation

Official Fedimg documentation can be found at RTFD or in docs/.

If you are interested in contributing to fedimg, you can read the developer documentation.

Triggering jobs

Fedimg is designed to perform automatically when it sees the appropriate fedmsg. However, sometimes, it's useful to be able to quickly trigger a job manually. If Fedimg is installed properly, the bin/trigger_upload.py script can be used for this purpose:

usage: trigger_upload.py [-h] -u URL -c COMPOSE_ID [-p]

Trigger a manual upload process with the specified raw.xz URL

optional arguments:
  -h, --help            show this help message and exit
  -u URL, --url URL     .raw.xz URL
  -c COMPOSE_ID, --compose-id COMPOSE_ID
                        compose id of the .raw.xz file
  -p, --push-notifications
                        Bool to check if we need to push fedmsg notifications

This script simply skips the part where Fedimg listens for the fedmsg, and allows you to directly provide a URL to a raw.xz image file that you'd like uploaded. Otherwise, Fedimg performs the same as during automatic operation.

Providers

We hope to simultaneously upload our cloud images to a variety of internal and external spaces. Currently, the code supports Amazon EC2. Work has begun toward supporting Rackspace, GCE, and HP. We're currently waiting on some legal developments to determine what sort of account and access we'll have to these providers.

Contributors

• David Gay [email protected]
• Ralph Bean [email protected]
• Sayan Chowdhury [email protected]

fedimg's Issues

NameError: global name 'LOG' is not defined

Callstack that lead to the logging statement
--------------------------------------------
```python
  File "/usr/lib64/python2.7/threading.py", line 785 in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib64/python2.7/threading.py", line 812 in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765 in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 167 in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 118 in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 81 in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py", line 185 in _work_loop
    self._do_work(message)
  File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py", line 209 in _do_work
    self.log.exception(message)

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py", line 206, in _do_work
    self.consume(message)
  File "/usr/lib/python2.7/site-packages/fedimg/consumers.py", line 88, in consume
    LOG.debug("%r is unsupported compose" % compose_id)
NameError: global name 'LOG' is not defined

IndexError when no raw.xz files found

For some time, we've been getting error logs when Fedimg tries to perform an upload with an unsupported image type, like Vagrant or Docker. I've been misunderstanding the problem for some time, and now realize that the problem was with a different part of the code that I was looking at. I'll have a quick fix for this in a few moments.

Please build eu-central (Frankfurt) AMI IDs again

Since a few releases we don't have the eu-central AMI ID anymore. It would be nice to have it again. I files also a releng ticket for it, just linking it here too.
https://fedorahosted.org/rel-eng/ticket/6259

CLI tool that accesses web API

Possibly develop a CLI tool that provides quick access to the data provided by the web API.

fedimg fails on Modular compose

Callstack that lead to the logging statement
--------------------------------------------
```python
  File "/usr/lib64/python2.7/threading.py", line 785 in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib64/python2.7/threading.py", line 812 in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765 in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 113 in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 65 in mapstar
    return map(*args)
  File "/usr/lib/python2.7/site-packages/fedimg/uploader.py", line 47 in <lambda>
    results = pool.map(lambda s: s.upload(compose_meta), services)
  File "/usr/lib/python2.7/site-packages/fedimg/services/ec2.py", line 566 in upload
    log.exception("Failure")

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fedimg/services/ec2.py", line 313, in upload
    "output: {2}".format(status, cmd, data))
EC2UtilityException: Problem writing image to utility instance volume. Command exited with status 1.
command: sudo sh -c 'curl -L http://kojipkgs.fedoraproject.org/compose/Fedora-Modular-Rawhide-20170824.n.0/compose/CloudImages/x86_64/images/Fedora-Cloud-Base-Rawhide-20170824.n.0.x86_64.raw.xz | xzcat > /dev/xvdb'
output:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
114   458  114   458    0     0   1723      0 --:--:-- --:--:-- --:--:--  6361
  0   458    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

Get autocloud to do the testing on AMIs

Right now fedimg does some dumb testing on its own, but we should separate this out.

Create instance-store-backed AMIs

Right now, Fedimg creates EBS-backed AMIs. We should also create instance-store-backed AMIs. The difference is that EBS-backed AMIs are launched with EBS which incurs slight additional charges whenever it is used, whereas instance-store backed AMIs will only incur the S3 storage charge to the AMI creators (us). We should provide both types of AMI.

If you're looking at this, feel free to tackle it. I'll report in if I start working on it, but I'm doing other things right now.

Docker integration

This is a "second half of the summer" thing, if I get to it by then. We want official Fedora images on the docker registry. These are actually "containers" which are a bit different than images. This will take a little research on my part as I don't have any experience with Docker outside of reading how it generally works and why it is useful. Any tips are appreciated.

• Docker images push to Docker registry

Add secondary cloud providers

Now that internal downloads are done, once I finish AMI uploads to each EC2 region, I'll want to move on to some other cloud providers which are of particular interest to Fedora. Feel free to reorder these services if you'd like me to prioritize.

• Google CE integration - TODO - check back with @kushaldas about legal status.
• Azure - TODO - check back with @mattdm about legal status.
• Digital Ocean (they do this work themselves, but maybe we can set up something to help automate it for them. we need to ping them). @sayanchowdhury will follow up on this one. Let's ask @dustymabe about it!

We used to some OpenStack sites here, but we decided that they're not worth focusing on (or they're dead, in some cases).

Automatic uploads to primary cloud image providers

Now that I've got a repo going, I'm going to move my task lists from my private Trello to GitHub.

This relates to https://fedorahosted.org/cloud/ticket/35

• Consume Koji build messages
• Get absolute image file URLs
• Download to internal Fedora FTP server
• Amazon EC2 AMIs in each region
  • ~~Utilize already-existing Fedora releng EC2 scripts to register AMIs from disk images (see https://git.fedorahosted.org/cgit/releng/tree/scripts/cloud) ~~AND/OR use ImageFactory code~~~~ (I'm writing my own code with apache-libcloud, but using older scripts for reference where appropriate
    • Decide on which script(s) to use
  • Clean up already-existing scripts to upload entire disk (i.e. /dev/sda instead of /dev/sda1, also, see https://fedorahosted.org/rel-eng/ticket/5881) This will be done by default in fedimg
  • releng ticket 5775 (https://fedorahosted.org/rel-eng/ticket/5775)

Following this ticket, we will need to start adding "secondary" priority providers, such as Rackspace.

No new AWS images for Fedora-Cloud-Base-25 in eu-west-1

There used to be a daily AWS images in the eu-west-1 region for Fedora-Cloud-Base-25 but that suddenly stopped after 2017-02-14. There are still daily images for Fedora-Atomic-25 being uploaded though.

It seems that all AWS regions are affected (I checked eu-west-1, eu-central-1 and us-east-1).

possibly produce virt-builder index.asc files?

See https://lists.fedoraproject.org/pipermail/cloud/2014-June/003950.html and particularly the reference to http://libguestfs.org/virt-builder.1.html -- maybe we could produce this automatically as part of the image upload?

(Not sure that this is the right place. Just wanted to note it as a possibility.)

Improve Exception Handling

fedimg at the moment is heavily dependent on third-party APIs. We need to have appropriate exception handling and raise exception whenever API fails.

Improve optional download progress indicator

The optional (enabled by default) progress indicator for downloading files has a couple appearance issues because I quickly hacked it together from code samples I found on the 'net. Suggested improvements are listed below.

• Use MB to measure data instead of bytes
• Eliminate strange cursor movement that occurs at every status 'tick'
  • Turns out this doesn't happen if the cursor isn't located in the same place as the progress indicator
• Perhaps include a progress bar?

Drop 'os' and 'ver' from the AWS_AMIS config

The config has a list of regions to upload to, which is called AWS_AMIs internally. It is a pipe-delimited list of regions with corresponding AMIs. For unknown reasons, each row contains an 'os' entry and a 'ver' entry (for example 'RHEL' and '6.5'). From what I can tell, they are not used. Can they just be removed?

use qcow2 instead of .raw.xz

See https://pagure.io/pungi-fedora/pull-request/381

There's no reason for releng to produce two formats. Anyone who wants raw can just do e.g.:

qemu-img convert -f qcow2 -O raw Fedora-Atomic-27-20170926.n.0.x86_64.qcow2 Fedora-Atomic-27-20170926.n.0.x86_64.raw

Image URL is gets sets to None for both image.upload & image.publish messages

The image_url in the fedmsg is None for the regions other than the base_region.

Example
https://apps.fedoraproject.org/datagrepper/id?id=2018-513e53ed-f5e8-4bc2-aaef-799c1c8ea1ad&is_raw=true&size=extra-large
https://apps.fedoraproject.org/datagrepper/id?id=2018-4406b0ff-3a7a-497f-a6b3-ac44a277a6f0&is_raw=true&size=extra-large

Don't use subprocess.Popen

fedimg is highly dependent on subprocess.Popen to execute commands.

• wget source image
• euca-import-volume
• euca-describe-conversion-tasks

This needs to migrate to alternative method to deprecate the use of subprocess.Popen

Store in database the Image to ami-id mapping

We can store the Image name, URL mapped to the AMI ids and Snapshot ids. It will be easier for us to track the images.

duplicate fedmsg publish for us-east-1

from the manual trigger we did yesterday I'm seeing two fedmsg published for us-east-1:

$ python ~/Downloads/get_ami.py 
Fedora-Cloud-Base-28-1.1.x86_64               eu-west-2            ami-ca14f7ad    hvm           gp2            
Fedora-Cloud-Base-28-1.1.x86_64               ap-northeast-1       ami-a600e6d9    hvm           gp2            
Fedora-Cloud-Base-28-1.1.x86_64               eu-central-1         ami-436640a8    hvm           gp2            
Fedora-Cloud-Base-28-1.1.x86_64               us-west-1            ami-77b3ae17    hvm           gp2            
Fedora-Cloud-Base-28-1.1.x86_64               us-west-2            ami-f5fc948d    hvm           gp2            
Fedora-Cloud-Base-28-1.1.x86_64               ap-southeast-2       ami-3f6ca65d    hvm           gp2            
Fedora-Cloud-Base-28-1.1.x86_64               ca-central-1         ami-c40e8fa0    hvm           gp2            
Fedora-Cloud-Base-28-1.1.x86_64               ap-southeast-1       ami-04c7e878    hvm           gp2            
Fedora-Cloud-Base-28-1.1.x86_64               sa-east-1            ami-9c8ad9f0    hvm           gp2            
Fedora-Cloud-Base-28-1.1.x86_64               ap-northeast-2       ami-a2ff56cc    hvm           gp2            
Fedora-Cloud-Base-28-1.1.x86_64               eu-west-1            ami-8b0824f2    hvm           gp2            
Fedora-Cloud-Base-28-1.1.x86_64               ap-south-1           ami-052a0b6a    hvm           gp2            
Fedora-Cloud-Base-28-1.1.x86_64               eu-west-2            ami-691af90e    hvm           standard       
Fedora-Cloud-Base-28-1.1.x86_64               ap-northeast-1       ami-8b05e3f4    hvm           standard       
Fedora-Cloud-Base-28-1.1.x86_64               eu-central-1         ami-6065438b    hvm           standard       
Fedora-Cloud-Base-28-1.1.x86_64               us-west-1            ami-25adb045    hvm           standard       
Fedora-Cloud-Base-28-1.1.x86_64               us-west-2            ami-56fb932e    hvm           standard       
Fedora-Cloud-Base-28-1.1.x86_64               ap-southeast-2       ami-976ea4f5    hvm           standard       
Fedora-Cloud-Base-28-1.1.x86_64               ca-central-1         ami-67129303    hvm           standard       
Fedora-Cloud-Base-28-1.1.x86_64               ap-southeast-1       ami-b8c5eac4    hvm           standard       
Fedora-Cloud-Base-28-1.1.x86_64               sa-east-1            ami-0e9dce62    hvm           standard       
Fedora-Cloud-Base-28-1.1.x86_64               ap-northeast-2       ami-ddf851b3    hvm           standard       
Fedora-Cloud-Base-28-1.1.x86_64               eu-west-1            ami-420d213b    hvm           standard       
Fedora-Cloud-Base-28-1.1.x86_64               ap-south-1           ami-c12b0aae    hvm           standard       
Fedora-Cloud-Base-28-1.1.x86_64               us-east-1            ami-e754e298    hvm           gp2            
Fedora-Cloud-Base-28-1.1.x86_64               us-east-1            ami-e754e298    hvm           gp2            
Fedora-Cloud-Base-28-1.1.x86_64               us-east-1            ami-5c69df23    hvm           standard       
Fedora-Cloud-Base-28-1.1.x86_64               us-east-1            ami-5c69df23    hvm           standard

Handle exception if compose type is unsupported

We shouldn't throw a traceback (and receive an email) for this:

UnsupportedComposeError: get_release(): fedfind does not support updates or updates-testing composes like Fedora-Epel-7-updates-20180410.0 as they contain no images

Web dashboard and API

Should provide image availability and location information.

• Deploy Pyramid
• Pull data from Datanommer
• Pretty web dashboard
• Public API

Taskotron integration

This will come down the road. It's probably at the end of the "main objectives" for this summer. When images are done uploading, Taskotron should make them available for test runs.

• Communicate with Tim Flink (feel free to link him if he has a GH account) to find out Taskotron status post-FAD and see what needs to happen next to get this ball rolling
• Overcome "chicken/egg" problem of having targets available in Taskotron
• Consider and document future opportunities for after the summer
• Taskotron integration fully working

fedimg seems to be stuck during initial AMI upload in us-east-1 region

From past two days, cloud AMIs upload for Fedora nightly composes seems to be stuck at uploading AMI to the very first region which is us-east-1.

Log messages say "[fedimg.uploader INFO] AWS EC2Service process is completed." but, it doesn't look like upload really got completed.

fedimg should cancel conversion task if import volume fails

If ImportVolume fails, then fedimg should make sure that all the conversion tasks which are not completed should be cancelled.

Missing dependency on python2-fedfind

python-fedimg-0.7-1.el7.noarch is missing a dependency on python2-fedimg, making it not start

Incorporate fedimg into the releng dash

This ticket encompasses adding appropriate fedimg data to the releng dash, and also sprucing the dash up a bit.

• Add status of cloud image uploads (started, success, failure)
• Add cloud image availability and location information
• Add status of tests run on image, as per #7
• Add a method to filter information types on the releng dash (i.e. so that people looking for say cloud image upload info don't have to scroll past other info)

Write supporting tools for sysadmins

we need some tools:

• we need something to list all the running instances in AWS (using the keys in the fedimg config)
• we need something to kill a single instance

Combining those two, we can clean up the AWS account when it gets congested. We can clean it from the CLI on fedimg01 instead of having to bug someone with rights to the AWS console to go and do it by hand (dgilmore).

fedimg needs to tag the AMIs

AMIs are uploaded on the nightly basis, both of the compose type production and nightly. It would be good to tag the AMIs appropriately in order to manage them. This would also help deprecating the old AMIs.

More info can be found here

• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html
• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html

Fix the Contributing.md

Contributing.md is out-dated and has broken links. The file needs an update with the Contribution guidelines and ways to contribute to fedimg.

Change hard-code 12 GB volume size

Right now, it looks like this code is always generating 12 GB volumes. Given that most of our images are only 3 GB, this seems like a waste of space & money. It should generally be possibly to inspect the size of the image (or even retrieve the information from the TDL file in the koji task) so that we don't have to guess the size.

list index out of range

Saw this come across the sysadmin-datanommer emails:

Message
-------
[2015-03-14 12:57:20][    fedmsg   ERROR]
{'body': {u'username': u'apache', u'certificate': u'LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVTakNDQTdPZ0F3SUJBZ0lDQWRzd0RRWUpL\nb1pJaHZjTkFRRUZCUUF3Z2FBeEN6QUpCZ05WQkFZVEFsVlQKTVFzd0NRWURWUVFJRXdKT1F6RVFN\nQTRHQTFVRUJ4TUhVbUZzWldsbmFERVhNQlVHQTFVRUNoTU9SbVZrYjNKaApJRkJ5YjJwbFkzUXhE\nekFOQmdOVkJBc1RCbVpsWkcxelp6RVBNQTBHQTFVRUF4TUdabVZrYlhObk1ROHdEUVlEClZRUXBF\nd1ptWldSdGMyY3hKakFrQmdrcWhraUc5dzBCQ1FFV0YyRmtiV2x1UUdabFpHOXlZWEJ5YjJwbFkz\nUXUKYjNKbk1CNFhEVEUxTURFd05USXhNemd5TVZvWERUSTFNREV3TWpJeE16Z3lNVm93Z2RneEN6\nQUpCZ05WQkFZVApBbFZUTVFzd0NRWURWUVFJRXdKT1F6RVFNQTRHQTFVRUJ4TUhVbUZzWldsbmFE\nRVhNQlVHQTFVRUNoTU9SbVZrCmIzSmhJRkJ5YjJwbFkzUXhEekFOQmdOVkJBc1RCbVpsWkcxelp6\nRXJNQ2tHQTFVRUF4TWlhMjlxYVMxcmIycHAKTURFdWNHaDRNaTVtWldSdmNtRndjbTlxWldOMExt\nOXlaekVyTUNrR0ExVUVLUk1pYTI5cWFTMXJiMnBwTURFdQpjR2g0TWk1bVpXUnZjbUZ3Y205cVpX\nTjBMbTl5WnpFbU1DUUdDU3FHU0liM0RRRUpBUllYWVdSdGFXNUFabVZrCmIzSmhjSEp2YW1WamRD\nNXZjbWN3Z1o4d0RRWUpLb1pJaHZjTkFRRUJCUUFEZ1kwQU1JR0pBb0dCQU9PaSt4VEsKSStQdEU2\nM!
 ll0ZkkzM3
 g4bW5UUE1ZU0hDZHB3NkhCNjZlbU1qeHRyMU95a082alY4NERQS1JDdzVFWCswUEV2\nbgozNFo2MCtKOXkxMS9JZzRuYTBhNlA5aVI5TlNXb1o3cENwQ3NMRUZIeFJUM0tqVlg4V3JkeURX\ndVRaSXNBN0c3CkVSL2wyRzlKaDBKSmVCNDBBcm9LMTJEeGllMjZ1dFQ5SHJpN0FnTUJBQUdqZ2dG\nWE1JSUJVekFKQmdOVkhSTUUKQWpBQU1DMEdDV0NHU0FHRytFSUJEUVFnRmg1RllYTjVMVkpUUVNC\nSFpXNWxjbUYwWldRZ1EyVnlkR2xtYVdOaApkR1V3SFFZRFZSME9CQllFRkxNVGpXekFPOURmYU83\nT1h0K2VkdFVMd2I3Vk1JSFZCZ05WSFNNRWdjMHdnY3FBCkZHdEFXdmtTQ0lsWjUxbmxCZlVDSFFw\nT2Z4UUFvWUdtcElHak1JR2dNUXN3Q1FZRFZRUUdFd0pWVXpFTE1Ba0cKQTFVRUNCTUNUa014RURB\nT0JnTlZCQWNUQjFKaGJHVnBaMmd4RnpBVkJnTlZCQW9URGtabFpHOXlZU0JRY205cQpaV04wTVE4\nd0RRWURWUVFMRXdabVpXUnRjMmN4RHpBTkJnTlZCQU1UQm1abFpHMXpaekVQTUEwR0ExVUVLUk1H\nClptVmtiWE5uTVNZd0pBWUpLb1pJaHZjTkFRa0JGaGRoWkcxcGJrQm1aV1J2Y21Gd2NtOXFaV04w\nTG05eVo0SUoKQU9OUUhrZFBGeDVGTUJNR0ExVWRKUVFNTUFvR0NDc0dBUVVGQndNQ01Bc0dBMVVk\nRHdRRUF3SUhnREFOQmdrcQpoa2lHOXcwQkFRVUZBQU9CZ1FCU2hrTnZ5ZzdvclZxbVVXdHdmZHpW\nUS92QnFXWkp1bys2a0RwckJIYUJlVWdmClA2RExEUEVPN3JHZXlxaFVtb0U3dm!
 RVRHBGeE5
 mbC9E\nemtBOHJBRWVsMjdGUXZ5RVFkUDJFaHpPUjZlTldwT0cKdEVJU0RML1piRFgwZzRrSWR5YjVoWGZY\nbUJzbzQ1RmdPYkNGUkdsTlg5SUU1Ly95bmxCZU5CMzEvYlk1bmc9PQotLS0tLUVORCBDRVJUSUZJ\nQ0FURS0tLS0tCg==\n', u'i': 1, u'timestamp': 1426337840, u'msg_id': u'2015-924858d6-fe96-4939-b15c-e3a029235b43', u'crypto': u'x509', u'topic': u'org.fedoraproject.prod.buildsys.task.state.change', u'signature': u'fDoQERRuiAjpIlWCSYMxGtcnXzOI1HkNEyTZTVROcyf5upz2HdCLzSsoqQNZMcANV8iMTmKD1dva\nTofPHKdg6vRs9j9AVCplcz1KJQY0cnWrWUB1ObmFbLl+x+/FaxtVMw/WUFuA1hMyc/9BCom86zpg\np9yrNbtCqoZVY7wB4nY=\n', u'msg': {u'info': {u'children': [{u'parent': 9233187, u'channel_id': 12, u'start_time': u'2015-03-14 12:48:22.037546', u'state': 2, u'awaited': False, u'label': u'x86_64', u'priority': 19, u'completion_time': u'2015-03-14 12:57:11.982569', u'waiting': None, u'create_time': u'2015-03-14 12:48:10.693851', u'id': 9233192, u'owner': 2745, u'host_id': 160, u'arch': u'x86_64', u'method': u'createImage'}], u'parent': None, u'channel_i!
 d': 12, u
 'start_time': u'2015-03-14 12:48:09.903671', u'request': [u'Fedora-Docker-Base', u'22', [u'x86_64'], u'f22-candidate', u'http://kojipkgs.fedoraproject.org/mash/branched-20150314/22/$arch/os/', {u'kickstart': u'work/cli-image/1426337265.628317.mihniGVK/fedora-docker-base-4276571.ks', u'disk_size': u'3', u'format': [u'docker'], u'scratch': True, u'repo': [u'http://kojipkgs.fedoraproject.org/mash/branched-20150314/22/$arch/os/'], u'release': u'20150314', u'distro': u'Fedora-20'}], u'state': 2, u'awaited': None, u'method': u'image', u'priority': 20, u'completion_time': u'2015-03-14 12:57:20.589813', u'waiting': False, u'create_time': u'2015-03-14 12:47:45.835651', u'result': u'Scratch image build results in: /mnt/koji/work/tasks/3192/9233192', u'owner': 2745, u'host_id': 160, u'label': None, u'arch': u'noarch', u'id': 9233187}, u'old': u'OPEN', u'attribute': u'state', u'method': u'image', u'instance': u'primary', u'owner': u'masher', u'new': u'CLOSED', u'srpm': u'Fedora-Docker-!
 Base', u'
 id': 9233187}}, 'topic': u'org.fedoraproject.prod.buildsys.task.state.change'}

Process Details
---------------
host:     fedimg01.phx2.fedoraproject.org
PID:      6489
name:     fedmsg-hub
command:  /usr/bin/python /usr/bin/fedmsg-hub

Callstack that lead to the logging statement
--------------------------------------------
  File "/usr/lib64/python2.7/threading.py", line 784 in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib64/python2.7/threading.py", line 811 in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 764 in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 167 in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 118 in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 81 in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py", line 192 in _work
    self.log.exception(message)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py", line 190, in _work
    self.consume(message)
  File "/usr/lib/python2.7/site-packages/fedimg/consumers.py", line 106, in consume
    fedimg.uploader.upload(self._get_upload_urls(builds))
  File "/usr/lib/python2.7/site-packages/fedimg/consumers.py", line 60, in _get_upload_urls
    upload_files.append(get_rawxz_url(task_result))
  File "/usr/lib/python2.7/site-packages/fedimg/util.py", line 52, in get_rawxz_url
    file_name = [f for f in task_result['files'] if f.endswith('.raw.xz')][0]
IndexError: list index out of range

"test_consumers.TestFedimgConsumer" fails due to network dependency

Relevant error log from trying to run that test:

AssertionError: Expected call: upload(compose_id='Fedora-Rawhide-20170708.n.0', pool=<ANY>, urls=['http://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20170708.n.0/compose/CloudImages/x86_64/images/Fedora-Cloud-Base-Rawhide-20170708.n.0.x86_64.raw.xz'])
Not called

....

fedfind.helpers: DEBUG: url_exists: checking https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20170708.n.0/compose as HTTP
fedfind.helpers: DEBUG: url_exists: checking https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20170708.n.0/compose as HTTP
fedfind.helpers: DEBUG: HTTP error! Retrying...
fedfind.helpers: DEBUG: Error: HTTP Error 404: Not Found

Presumably when it was written, koji was serving something there, but no longer is. Unit tests shouldn't be reaching out to the network, so that get_release call (I think that's what doing the network traffic) should probably be mocked too

Reorganize the way we upload images to AWS

We start up a helper image and then download the image we really want, and write it out to a snapshot.

Then we copy the image to all the other regions.

Apparently, according to gholms, we can write the image directly to s3 which should allow us to skip that first step.

Add the Atomic & Cloud variant for F28 onward releases.

Till F27, both cloud-base and atomic images were available under variant CloudImages. With F28 and onward releases, cloud-base image compose moved to cloud variant and atomic images moved under atomic variant.

we need to uncompress raw.xz files before creating AMIs out of them

We're getting reports of instances that aren't able to boot. I inspected one of these AMIs and it turns out the data on the volume is xz compressed data. We need to decompress the data before creating a volume/snapshot/AMI out of it. I imagine it would happen somewhere around here in the code.

Just to verify I took the xz compressed data from one of these AMIs, decompressed it, and wrote it back out to the volume and then booted the same instance using the uncompressed disk.

Fedimg is unable to process Fedora 28 Atomic Two Week compose

We recently started running compose for F28 Two Week . It seems fedimg fail to process it due to following error from log:

File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py", line 206, in _do_work
fedimg01 fedmsg-hub: self.consume(message)
File "/usr/lib/python2.7/site-packages/fedimg/consumers.py", line 105, in consume
images_meta.extend(get_value_from_dict(
AttributeError: 'NoneType' object has no attribute 'extend'

This is happening because, we don't have cloud-base images to process in F28 Two Week compose.

Remove scratch-task based uploading

According to this releng ticket it should be dead soon.

I added build-based uploading in #47, but didn't remove the scratch-task based uploading yet. That should be done, just to clean the code and keep it sane.

failures when uploading to us-east-2 and eu-west-3 regions

Not sure why but those two regions don't work for fedimg uploads.

[2018-04-26 00:25:22][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to us-east-2 started
[2018-04-26 00:25:23][fedimg.services.ec2.ec2imgpublisher    INFO] Could not register with name: u'Fedora-Cloud-Base-28-1.1.x86_64-hvm-us-east-2-standard-0'
[2018-04-26 00:25:23][fedimg.services.ec2.ec2imgpublisher    INFO] Failed
[2018-04-26 00:25:23][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to sa-east-1 started
[2018-04-26 00:25:26][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to sa-east-1 is completed.
[2018-04-26 00:25:26][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to ap-southeast-1 started
[2018-04-26 00:25:54][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to ap-southeast-1 is completed.
[2018-04-26 00:25:54][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to ca-central-1 started
[2018-04-26 00:26:03][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to ca-central-1 is completed.
[2018-04-26 00:26:03][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to ap-southeast-2 started
[2018-04-26 00:26:23][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to ap-southeast-2 is completed.
[2018-04-26 00:26:23][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to us-west-2 started
[2018-04-26 00:26:25][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to us-west-2 is completed.
[2018-04-26 00:26:25][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to us-west-1 started
[2018-04-26 00:26:35][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to us-west-1 is completed.
[2018-04-26 00:26:35][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to eu-central-1 started
[2018-04-26 00:26:44][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to eu-central-1 is completed.
[2018-04-26 00:26:44][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to ap-northeast-1 started
[2018-04-26 00:26:51][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to ap-northeast-1 is completed.
[2018-04-26 00:26:51][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to eu-west-2 started
[2018-04-26 00:27:10][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to eu-west-2 is completed.
[2018-04-26 00:27:10][fedimg.services.ec2.ec2imgpublisher    INFO] Copy ami-5c69df23 to eu-west-3 started
[2018-04-26 00:27:10][fedimg.services.ec2.ec2imgpublisher    INFO] Could not register with name: u'Fedora-Cloud-Base-28-1.1.x86_64-hvm-eu-west-3-standard-0'
[2018-04-26 00:27:10][fedimg.services.ec2.ec2imgpublisher    INFO] Failed
[2018-04-26 00:27:10][fedimg.services.ec2.ec2imgpublisher    INFO] Publish image (ami-c12b0aae) in ap-south-1 started

Finalize fedimg meta code

Need to finalize fedimg meta details in this repo: https://github.com/fedora-infra/fedmsg_meta_fedora_infrastructure

Current work is sitting here until I finish it and get some +1s on merging it in to develop/master: https://github.com/fedora-infra/fedmsg_meta_fedora_infrastructure/tree/feature/fedimg-meta

README 'trigger_upload' instructions out of date

The readme instructions claim I can run ./bin/trigger_upload.py SOME_RAWXZ_URL to upload things.

That script still has from fedimg.util import virt_types_from_url even though that import was changed, and even after fixing the import issues, the script still doesn't run.

How is fedimg typically run? Should trigger_upload be fixed or deleted?

Spin up an instance and run a simple test after cloud image upload

Adding an idea from @mattdm:

After cloud image is uploaded:

• a test instance is booted in any one of our available environments (EC2, Fedora cloud, whatever is easiest)
• a simple test is run against that instance
  • connect via ssh and run /bin/true
  • it should be easy to sub in any other "test command" (more complicated tests can come later)
• results of that test are reported on fedmsg (and possibly by email to cloud-sig list)
• make sure test instance has been terminated and its resources released

show command error output when commands fail

In the code

fedimg/fedimg/utils.py

Lines 101 to 109 in 3946ed1

	def external_run_command(command):
	_log.debug("Starting the command: %r" % command)
	ret = subprocess.Popen(' '.join(command), stdin=subprocess.PIPE, shell=True,
	stdout=subprocess.PIPE, stderr=subprocess.PIPE,
	close_fds=True)
	out, err = ret.communicate()
	_log.debug("Finished executing the command: %r" % command)
	retcode = ret.returncode
	return out, err, retcode

we need to give the user the error message in the logs when something fails. This mainly stems from

fedimg/fedimg/utils.py

Lines 124 to 140 in 3946ed1

	def get_source_from_image(image_url):
	tmpdir = tempfile.mkdtemp()
	file_name = get_file_name_image(image_url)
	file_path = os.path.join(tmpdir, file_name)
	_log.info("[PREP] Preparing temporary directory for download: %r" % tmpdir)
	output, error, retcode = external_run_command([
	'wget',
	image_url,
	'-P',
	tmpdir
	])
	if retcode != 0:
	return ''
	return file_path

where the code doesn't tell you anything about what failed, just returns ''

Add generic re-try logic to all of fedimg's operations

Right now, the cloud just plain fails sometimes. Like this error email I got today:

[2015-09-17 13:20:12][    fedmsg   ERROR]
Image copy to ap-southeast-2 failed


Process Details
---------------
host:     fedimg01.phx2.fedoraproject.org
PID:       8640
name:     fedmsg-hub
command:  /usr/bin/python /usr/bin/fedmsg-hub

Callstack that lead to the logging statement
--------------------------------------------
  File "/usr/lib64/python2.7/threading.py", line 784 in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib64/python2.7/threading.py", line 811 in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 764 in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 113 in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 65 in mapstar
    return map(*args)
  File "/usr/lib/python2.7/site-packages/fedimg/uploader.py", line 46 in <lambda>
    results = pool.map(lambda s: s.upload(), services)
  File "/usr/lib/python2.7/site-packages/fedimg/services/ec2.py", line 627 in upload
    ami['region']))
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fedimg/services/ec2.py", line 605, in upload
    description=self.image_desc)
  File "/usr/lib/python2.7/site-packages/libcloud/compute/drivers/ec2.py", line 2691, in copy_image
    self.connection.request(self.path, params=params).object)
  File "/usr/lib/python2.7/site-packages/libcloud/common/base.py", line 799, in request
    response = responseCls(**kwargs)
  File "/usr/lib/python2.7/site-packages/libcloud/common/base.py", line 143, in __init__
    headers=self.headers)
BaseHTTPError: InternalError: An internal error has occurred

The cloud just doesn't always work 100% of the time; we know that.

So, we should wrap most of fedimg's operations in little re-try loops that try 3 or 4 times before giving up.

Allow an option for HVM image registration

The ec2 module currently omits the virtualization_type option in image registration, which means that HVM images cannot be registered. This is currently blcoked because libcloud also omits this parameter.

Python 3 Migrations

Python 2 deprecation is near. So, fedimg needs to be ported to Python 3.

fedimg doesn't remove images after upload

on fedimg01.phx2.fedoraproject.org:

du -sh /tmp

13G /tmp

all these are dirs with images, some of which I deleted to get space under control.

ls -lR tmpjI3dm0
tmpjI3dm0:
total 160400
-rw-r--r--. 1 fedmsg fedmsg 164249592 Apr 14 13:02 Fedora-Cloud-Base-28-20180414.n.0.x86_64.raw.xz

make sure fedimg cleans up after it uploads (or fails to).

Add scripts to remove old AMIs we have uploaded

We need a script to run as cron that will remove old AMIs we have uploaded.

• We should keep the last month of AMIs, generally.
• We should keep the GA final release AMIs forever.
• The Alpha, Beta, and TC RC releases might should be kept for a month or two.
• All other AMIs should be deleted after they've been up for a month.

Generate a work report at the end of a job

It would be cool if a human-readable, nicely-formatted job report was written to /var/tmp/. It'd be cool for archival purposes, but especially when triggering manual upload jobs. Shouldn't be hard to do at all.

Fedimg Documentation

The documentation for the code base needs to be changed after the 1.0.0 release. [Fedimg Read the Docs ] (http://fedimg.readthedocs.io/en/latest/) should also be updated to explaining the new work flow, how to use the scripts, and managing fedimg.