ceph / chacra Goto Github PK

View Code? Open in Web Editor NEW

9.0 78.0 16.0 905 KB

A binary/file REST API to aid in multi-distro|arch|release management

Python 88.02% Shell 6.88% Mako 0.36% Jinja 4.74%

chacra's People

Contributors

Stargazers

Watchers

Forkers

andrewschoen christinameno joehandzik jdandrea pombredanne tchaikov toabctl sravikaz isabella232 dmick ibm-s390-cloud sine-cloud yanghonggang guits ljflores cephsea

chacra's Issues

auto-sign repos (or packages)

Some projects need to have automatic repository (for DEBs) or package (for RPMs) signing.

An unattended signing process requires a number of constraints, primarily is the ability to not require a passphrase (or to have it in clear text somewhere).

implement metrics

These should fire to statsd that will end up relaying to a central Graphite instance.

Things to do:

wrapper to automatically go on the right bucket based on hostname+module+metric
configuration settings and docs to properly set it up
deployment tasks to create a local statsd instance that would run alongside this service

Repos in shaman and chacra is NULL while Finished is SUCCESS?

I make a run.py just like "shaman/config/run.py" to run chacra.
After the Jenkins build finished with success, the Repos in shaman and chacra is NULL. Would you tell me why is this or how to start (setup) chacra. Thank you. @andrewschoen @alfredodeza
The following is part of log:

cat /home/jenkins/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/trusty/DIST/trusty/repo-extra.json
{
"version":"12.0.0-810-gda06227",
"package_manager_version":"12.0.0-810-gda06227-1trusty",
"build_url":"http://jenkins.star.com.cn/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=trusty,DIST=trusty/97/",
"root_build_cause":"MANUALTRIGGER",
"node_name":"phy-10.118.203.10-ubuntu-trusty",
"job_name":"ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=trusty,DIST=trusty"
}
cat /home/ubuntu/.chacractl
url = "http://192.168.34.252/"
user = "admin"
key = "secret"
ssl_verify = False
curl -X POST -H Content-Type:application/json --data @/home/jenkins/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/trusty/DIST/trusty/repo-extra.json -u admin:secret http://192.168.34.252/repos/ceph/master/da06227c7fdd71936fc152e29551268e78541e21/ubuntu/trusty/flavors/default/extra/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 1198 100 771 100 427 15724 8708 --:--:-- --:--:-- --:--:-- 16062
{"project_name": "ceph", "extra": {"build_url": "http://jenkins.star.com.cn/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=trusty,DIST=trusty/97/", "root_build_cause": "MANUALTRIGGER", "version": "12.0.0-810-gda06227", "node_name": "phy-10.118.203.10-ubuntu-trusty", "job_name": "ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=trusty,DIST=trusty", "package_manager_version": "12.0.0-810-gda06227-1trusty"}, "distro_version": "trusty", "path": null, "flavor": "default", "size": 0, "needs_update": true, "sha1": "da06227c7fdd71936fc152e29551268e78541e21", "is_queued": false, "is_updating": false, "type": "deb", "modified": "2017-02-27 17:09:20.166290", "signed": false, "ref": "master", "archs": ["x86_64"], "distro": "ubuntu"}+ /tmp/venv.Dp0IvzuZV8/bin/chacractl repo update ceph/master/da06227c7fdd71936fc152e29551268e78541e21/ubuntu/trusty/flavors/default
[requests.packages.urllib3.connectionpool][DEBUG ] Starting new HTTP connection (1): 192.168.34.252
[requests.packages.urllib3.connectionpool][DEBUG ] http://192.168.34.252:80 "HEAD /repos/ceph/master/da06227c7fdd71936fc152e29551268e78541e21/ubuntu/trusty/flavors/default/update HTTP/1.1" 200 0
[chacractl.api.repos][INFO ] POST: http://192.168.34.252/repos/ceph/master/da06227c7fdd71936fc152e29551268e78541e21/ubuntu/trusty/flavors/default/update
[requests.packages.urllib3.connectionpool][DEBUG ] Starting new HTTP connection (1): 192.168.34.252
[requests.packages.urllib3.connectionpool][DEBUG ] http://192.168.34.252:80 "POST /repos/ceph/master/da06227c7fdd71936fc152e29551268e78541e21/ubuntu/trusty/flavors/default/update HTTP/1.1" 200 771
[chacractl.api.repos][INFO ] needs_update: True
[chacractl.api.repos][INFO ] project_name: ceph
[chacractl.api.repos][INFO ] is_queued: False
[chacractl.api.repos][INFO ] is_updating: False
[chacractl.api.repos][INFO ] extra: {u'build_url': u'http://jenkins.star.com.cn/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=trusty,DIST=trusty/97/', u'root_build_cause': u'MANUALTRIGGER', u'version': u'12.0.0-810-gda06227', u'node_name': u'phy-10.118.203.10-ubuntu-trusty', u'job_name': u'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=trusty,DIST=trusty', u'package_manager_version': u'12.0.0-810-gda06227-1trusty'}
[chacractl.api.repos][INFO ] ref: master
[chacractl.api.repos][INFO ] modified: 2017-02-28 09:00:31.405657
[chacractl.api.repos][INFO ] distro_version: trusty
[chacractl.api.repos][INFO ] sha1: da06227c7fdd71936fc152e29551268e78541e21
[chacractl.api.repos][INFO ] signed: False
[chacractl.api.repos][INFO ] path: None
[chacractl.api.repos][INFO ] flavor: default
[chacractl.api.repos][INFO ] distro: ubuntu
[chacractl.api.repos][INFO ] type: deb
[chacractl.api.repos][INFO ] archs: [u'x86_64']
[chacractl.api.repos][INFO ] size: 0
echo Check the status of the repo at: http://192.168.34.254/api/repos/ceph/master/da06227c7fdd71936fc152e29551268e78541e21/ubuntu/trusty/flavors/default/
Check the status of the repo at: http://192.168.34.254/api/repos/ceph/master/da06227c7fdd71936fc152e29551268e78541e21/ubuntu/trusty/flavors/default/
update_build_status completed ceph ubuntu trusty x86_64
state=completed
project=ceph
distro=ubuntu
distro_version=trusty
distro_arch=x86_64
submit_build_status POST completed ceph ubuntu trusty x86_64
http_method=POST
state=completed
project=ceph
distro=ubuntu
distro_version=trusty
distro_arch=x86_64
cat
cat
SHAMAN_URL=http://192.168.34.254/api/builds/ceph/
curl -X POST -H Content-Type:application/json --data @/home/jenkins/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/trusty/DIST/trusty/build_status.json -u admin:secret http://192.168.34.254/api/builds/ceph/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 772 100 2 100 770 136 52441 --:--:-- --:--:-- --:--:-- 55000
{}+ rm -fr /tmp/install-deps.12694
Notifying upstream projects of job completion
Finished: SUCCESS

when calling reprepro, Popen eats up stderr

This should be logged, especially when there are non-zero return codes

fix 127.0.0.1 issue for local connections in postgresql

To avoid:

Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?

deploy chacra failed

hosts file is

[test]
chacra1.xsky.com short_hostname=chacra1

deply.yml is

- hosts: test
  user: root
  roles:
    - common
    - repos
    - statsd
  vars:
     app_name: "chacra"
     use_ssl: true
     wsgi_file: wsgi.py
     wsgi_callable: application
     ansible_ssh_port: 22
     binary_root: "/opt/binaries"
     repos_root: "/opt/repos"
     branch: "master"
     development_server: true
     purge_repos: True
     fqdn: "{{ inventory_hostname }}"
     # required for first deploy
     api_user: "admin"
     api_key: "secret"
     # graphite reporting for statsd
     graphite_host: "3.shaman.xsky.com"
     # required
     graphite_api_key: '1234-asdf-1234'
     # callbacks
     callback_url: "https://3.shaman.xsky.com/api/repos/"
     # required for first deploy
     callback_user: "admin"
     callback_key: "secret"
     callback_verify_ssl: False
     health_ping: true
     health_ping_url: "https://3.shaman.xsky.com/api/nodes/"
     # use_letsencrypt: True
     # nginx_ssl_cert_path: "/etc/letsencrypt/live/{{ fqdn }}/fullchain.pem"
     # nginx_ssl_key_path: "/etc/letsencrypt/live/{{ fqdn }}/privkey.pem"
     nginx_ssl_cert_path: "/etc/ssl/certs/{{ fqdn }}-bundled.crt"
     nginx_ssl_key_path: "/etc/ssl/private/{{ fqdn }}.key"
     combine_deb_repos: False

The deploy command is 'ansible-playbook -vvv -i hosts deploy.yml'

it's failed with

TASK [common : populate the database for chacra] ***************************************************************************************************************************************************************************************
task path: /opt/chacra/src/chacra/deploy/playbooks/roles/common/tasks/postgresql.yml:81
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/commands/command.py
<chacra1.xsky.com> ESTABLISH SSH CONNECTION FOR USER: root
<chacra1.xsky.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/d61856f8b9 chacra1.xsky.com '/bin/sh -c '"'"'ALEMBIC_CONFIG=/opt/chacra/src/chacra/alembic-prod.ini /usr/bin/python && sleep 0'"'"''
<chacra1.xsky.com> (1, '\n{"changed": true, "end": "2021-08-06 16:34:13.372460", "stdout": "==> LOADING ENVIRONMENT\\nbasepath=/opt/chacra/src/chacra,module_name=prod", "cmd": ["/opt/chacra/bin/pecan", "populate", "/opt/chacra/src/chacra/prod.py"], "failed": true, "delta": "0:00:02.304443", "stderr": "Traceback (most recent call last):\\n  File \\"/opt/chacra/bin/pecan\\", line 8, in <module>\\n    sys.exit(CommandRunner.handle_command_line())\\n  File \\"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\\", line 96, in handle_command_line\\n    runner.run(sys.argv[1:])\\n  File \\"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\\", line 91, in run\\n    self.commands[ns.command_name]().run(ns)\\n  File \\"/opt/chacra/lib/python2.7/site-packages/chacra/commands/populate.py\\", line 34, in run\\n    self.load_app()\\n  File \\"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\\", line 162, in load_app\\n    return load_app(self.args.config_file)\\n  File \\"/opt/chacra/lib/python2.7/site-packages/pecan/core.py\\", line 213, in load_app\\n    set_config(config, overwrite=True)\\n  File \\"/opt/chacra/lib/python2.7/site-packages/pecan/configuration.py\\", line 253, in set_config\\n    config = conf_from_file(config)\\n  File \\"/opt/chacra/lib/python2.7/site-packages/pecan/configuration.py\\", line 183, in conf_from_file\\n    *imp.find_module(module_name, [basepath])\\n  File \\"/opt/chacra/src/chacra/prod.py\\", line 5, in <module>\\n    from prod_db import sqlalchemy\\nImportError: No module named prod_db", "rc": 1, "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "/opt/chacra/bin/pecan populate /opt/chacra/src/chacra/prod.py", "removes": null, "creates": null, "chdir": null, "stdin": null}}, "start": "2021-08-06 16:34:11.068017", "msg": "non-zero return code"}\n', '')
fatal: [chacra1.xsky.com]: FAILED! => {
    "changed": true, 
    "cmd": [
        "/opt/chacra/bin/pecan", 
        "populate", 
        "/opt/chacra/src/chacra/prod.py"
    ], 
    "delta": "0:00:02.304443", 
    "end": "2021-08-06 16:34:13.372460", 
    "invocation": {
        "module_args": {
            "_raw_params": "/opt/chacra/bin/pecan populate /opt/chacra/src/chacra/prod.py", 
            "_uses_shell": false, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "stdin": null, 
            "warn": true
        }
    }, 
    "msg": "non-zero return code", 
    "rc": 1, 
    "start": "2021-08-06 16:34:11.068017", 
    "stderr": "Traceback (most recent call last):\n  File \"/opt/chacra/bin/pecan\", line 8, in <module>\n    sys.exit(CommandRunner.handle_command_line())\n  File \"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\", line 96, in handle_command_line\n    runner.run(sys.argv[1:])\n  File \"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\", line 91, in run\n    self.commands[ns.command_name]().run(ns)\n  File \"/opt/chacra/lib/python2.7/site-packages/chacra/commands/populate.py\", line 34, in run\n    self.load_app()\n  File \"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\", line 162, in load_app\n    return load_app(self.args.config_file)\n  File \"/opt/chacra/lib/python2.7/site-packages/pecan/core.py\", line 213, in load_app\n    set_config(config, overwrite=True)\n  File \"/opt/chacra/lib/python2.7/site-packages/pecan/configuration.py\", line 253, in set_config\n    config = conf_from_file(config)\n  File \"/opt/chacra/lib/python2.7/site-packages/pecan/configuration.py\", line 183, in conf_from_file\n    *imp.find_module(module_name, [basepath])\n  File \"/opt/chacra/src/chacra/prod.py\", line 5, in <module>\n    from prod_db import sqlalchemy\nImportError: No module named prod_db", 
    "stderr_lines": [
        "Traceback (most recent call last):", 
        "  File \"/opt/chacra/bin/pecan\", line 8, in <module>", 
        "    sys.exit(CommandRunner.handle_command_line())", 
        "  File \"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\", line 96, in handle_command_line", 
        "    runner.run(sys.argv[1:])", 
        "  File \"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\", line 91, in run", 
        "    self.commands[ns.command_name]().run(ns)", 
        "  File \"/opt/chacra/lib/python2.7/site-packages/chacra/commands/populate.py\", line 34, in run", 
        "    self.load_app()", 
        "  File \"/opt/chacra/lib/python2.7/site-packages/pecan/commands/base.py\", line 162, in load_app", 
        "    return load_app(self.args.config_file)", 
        "  File \"/opt/chacra/lib/python2.7/site-packages/pecan/core.py\", line 213, in load_app", 
        "    set_config(config, overwrite=True)", 
        "  File \"/opt/chacra/lib/python2.7/site-packages/pecan/configuration.py\", line 253, in set_config", 
        "    config = conf_from_file(config)", 
        "  File \"/opt/chacra/lib/python2.7/site-packages/pecan/configuration.py\", line 183, in conf_from_file", 
        "    *imp.find_module(module_name, [basepath])", 
        "  File \"/opt/chacra/src/chacra/prod.py\", line 5, in <module>", 
        "    from prod_db import sqlalchemy", 
        "ImportError: No module named prod_db"
    ], 
    "stdout": "==> LOADING ENVIRONMENT\nbasepath=/opt/chacra/src/chacra,module_name=prod", 
    "stdout_lines": [
        "==> LOADING ENVIRONMENT", 
        "basepath=/opt/chacra/src/chacra,module_name=prod"
    ]
}

RUNNING HANDLER [common : restart app] *************************************************************************************************************************************************************************************************

PLAY RECAP *****************************************************************************************************************************************************************************************************************************
chacra1.xsky.com           : ok=31   changed=7    unreachable=0    failed=1

purge repos is not removing the repository directory path

It currently removes:

the actual binary
the binary model object (database)
repository model object (database)

But it doesn't remove the actual repository path (where the built repository goes to). This is trivial to fix as the repository model object has the path stored in it.

Provide mapping from a debian distros versions and codenames

For example, these two urls should provide the same output:

https://chacra.ceph.com/binaries/ceph/jewel/HEAD/ubuntu/xenial/ https://chacra.ceph.com/binaries/ceph/jewel/HEAD/ubuntu/16.04/

I don't think we care about this for rpm-based distros.

failing to get rabbitmq installed before enabling the service

This probably hasn't showed up in production because rabbitmq-server is already installed.

TASK [common : ensure chacra-celerybeat is enabled and running] ****************
fatal: [node1]: FAILED! => {"changed": false, "failed": true, "msg": "Failed to start chacra-celerybeat.service: Unit rabbitmq-server.service not found.\n"}

allow disabling of automatic repo creation

The current docs say it can be done but that really disables all repo creation (docs should get updated too)

Automatic repo creation is triggered when a new binary is uploaded. Projects that upload multiple binaries will benefit from not pilling up on repo creation requests.

Validation for distro and distro_version

Provide some simple validation for distro and distro_version. For example, if 'centos' is a known and accepted distro and that '7' is a valid distro_version for centos.

increase timeouts to 20 minutes

Because debug binaries are gigantic and it takes a while to POST them.

[chacractl.api.binaries][INFO  ] POSTing file: /extra/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/release/10.2.2/rpm/el7/RPMS/aarch64/ceph-debuginfo-10.2.2-0.el7.aarch64.rpm
[requests.packages.urllib3.connectionpool][INFO  ] Starting new HTTPS connection (1): chacra.ceph.com
[requests.packages.urllib3.connectionpool][DEBUG ] "POST /binaries/ceph/jewel/centos/7/arm64/ HTTP/1.1" 502 166
[chacractl.api.binaries][WARNING] 502 -> <html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

Traceback (most recent call last):
  File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/bin/chacractl", line 6, in <module>
    main.ChacraCtl()
  File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/lib/python2.7/site-packages/chacractl/main.py", line 38, in __init__
    self.main(argv)
  File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/lib/python2.7/site-packages/chacractl/decorators.py", line 68, in newfunc
    return f(*a, **kw)
  File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/lib/python2.7/site-packages/chacractl/main.py", line 82, in main
    parser.dispatch()
  File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/lib/python2.7/site-packages/tambo/dispatcher.py", line 21, in dispatch
    return instance.main()
  File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/lib/python2.7/site-packages/chacractl/api/binaries.py", line 127, in main
    self.post(url, filename)
  File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/lib/python2.7/site-packages/chacractl/api/binaries.py", line 82, in post
    response.raise_for_status()
  File "/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/DIST/centos7/MACHINE_SIZE/huge/venv/lib/python2.7/site-packages/requests/models.py", line 844, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://chacra.ceph.com/binaries/ceph/jewel/centos/7/arm64/

search broken?

Is search working? the following URL returns an empty list:

https://chacra.ceph.com/search/?name=ceph-deploy

test repos should not combine debian versions

For our chacra gitbuilder nodes we do not want to combine debian versions into one repo. We need to remove this configuration from all test chacra nodes: https://github.com/ceph/chacra/blob/master/deploy/playbooks/roles/common/templates/prod.py.j2#L116

In the example we found it also mixed two different sha1s into the same repo: https://2.chacra.ceph.com/r/ceph/master/b8c203fb64c80fe52a4d17562ddef39371347d42/ubuntu/xenial/flavors/default/pool/main/c/ceph/

prevent further pilling by marking repos as 'is_queued'

So that the loop that keeps adding repos to the processing queue will not add repos that are already present in the queue.

PUT requests for updates on binaries looks for the file in the wrong place

It uses: contents = request.POST.get('file', False) which is wrong because it is a PUT request, not a POST

handle large files better

Currently chacra reads an entire binary into memory and then writes it to a file. Some of Ceph's binaries are huge (ceph-test-dbg is 770MB) and this can easily overwhelm a chacra server so that it is unresponsive (SSH times out, etc).

From chacra/controllers/binaries/__init__.py:

        destination = os.path.join(dir_path, self.binary_name)
        with open(destination, 'wb') as f:
            try:
                f.write(file_obj.getvalue())
            except AttributeError:
                f.write(file_obj.read())

Instead of prefering f.write(file_obj.getvalue()), it would be good to prefer the file_obj.read() iterator (If that really works?) and make that read/write something like 8192 bits at a time.

properly restart celery and celery beat always always always

When deploying, to avoid the idiotic database connection error

statsd wrapper library

So that it can be used similarly to the logging module, configurable with a secret key and automatic host detection.

callback system

From @alfredodeza on May 24, 2016 18:59

Whenever a repository is fully built and ready to be consumed, it would be nice to be able to define a URL to get a POST with metadata information about the repository:

distro
distro version(s)
ref (and sha1?)
architecture
timestamp(s)

This would help if a separate system is in charge of understanding what has been built. For testing purposes it is crucial to be able to know what distro versions at what sha1 are ready to be consumed.

Copied from original issue: ceph/mita#74

`[..]/$arch/flavors/$flavor` endpoint

Implement a distinct flavors endpoint to accommodate different kinds of binaries for the same kind of build.

This should be optional, and the default would be to show the non-distinct flavor (which would help in any instance that doesn't care about build "flavors")

health-check/ping

Using a similar solution for the callback system (see issue #110) implement a "ping" request to a remote destination so that when an instance fails it can be removed from rotation.

Configurable options should look like:

ping_frequency (in seconds). Note it is critical to allow enough time in between checks on the receiving end to avoid being marked as down.
ping_url: the remote host and endpoint that receives the request
ping_retries: Retry count for failed ping requests (non 200 responses back)

The code should determine what constitutes a working system for its various systems: database, rabbitmq, http)

Docs for adding new chacra nodes

We need documentation for adding new chacra nodes to shaman.

ensure celery and celerybeat are enabled and running

For systemd when deploying.

Problem with ubuntu repos and/or package_manager_version

As I'm working on teuthology support for shaman/chacra, I've mostly been running tests against centos 7 since there aren't any ubuntu 14.04 builds yet. Today I tried a few tests against 16.04 and am seeing issues.

First, the rest runs. This one uses my wip branch but is using gitbuilder:
http://pulpito.ceph.com/zack-2016-09-09_08:40:06-smoke-jewel-distro-basic-vps/
This run uses shaman:
http://pulpito.ceph.com/zack-2016-09-09_08:39:27-smoke-jewel-distro-basic-vps/

Focusing on one job, it appears there may something wrong with the repos being generated, and package_manager_version isn't matching what the packages are actually using:

2016-09-09T14:55:05.362 INFO:teuthology.orchestra.run.vpm107:Running: u'echo deb https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default/ ubuntu/16.04 main | sudo tee /etc/apt/sources.list.d/ceph.list'
2016-09-09T14:55:05.382 INFO:teuthology.orchestra.run.vpm107.stdout:deb https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default/ ubuntu/16.04 main
2016-09-09T14:55:05.383 INFO:teuthology.orchestra.run.vpm107:Running: 'sudo apt-get update'
2016-09-09T14:55:05.666 INFO:teuthology.orchestra.run.vpm107.stdout:Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB]
2016-09-09T14:55:05.759 INFO:teuthology.orchestra.run.vpm107.stdout:Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
2016-09-09T14:55:05.808 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04 InRelease
2016-09-09T14:55:05.830 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04 Release
2016-09-09T14:55:05.839 INFO:teuthology.orchestra.run.vpm107.stdout:Hit:5 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
2016-09-09T14:55:05.853 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:05.875 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:05.897 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:05.919 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:05.920 INFO:teuthology.orchestra.run.vpm107.stdout:Hit:10 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
2016-09-09T14:55:05.941 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:05.963 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:05.984 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:06.006 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:06.028 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:06.050 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:06.072 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:06.094 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:06.116 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:06.138 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:06.160 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:06.182 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:06.204 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:06.226 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:06.248 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:06.270 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:06.292 INFO:teuthology.orchestra.run.vpm107.stdout:Err:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:06.292 INFO:teuthology.orchestra.run.vpm107.stdout:  404  Not Found
2016-09-09T14:55:06.314 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:06.337 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:06.359 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:06.397 INFO:teuthology.orchestra.run.vpm107.stdout:Fetched 94.5 kB in 0s (118 kB/s)
2016-09-09T14:55:07.397 INFO:teuthology.orchestra.run.vpm107.stdout:Reading package lists...
2016-09-09T14:55:07.439 INFO:teuthology.orchestra.run.vpm107.stderr:W: The repository 'https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04 Release' does not have a Release file.
2016-09-09T14:55:07.439 INFO:teuthology.orchestra.run.vpm107.stderr:E: Failed to fetch https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default/dists/ubuntu/16.04/main/binary-amd64/Packages  404  Not Found
2016-09-09T14:55:07.440 INFO:teuthology.orchestra.run.vpm107.stderr:E: Some index files failed to download. They have been ignored, or old ones used instead.
2016-09-09T14:55:07.441 INFO:teuthology.orchestra.run.vpm107:Running: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=10.2.2-508-g9bfc0cf-1xenial ceph-mds=10.2.2-508-g9bfc0cf-1xenial ceph-common=10.2.2-508-g9bfc0cf-1xenial ceph-fuse=10.2.2-508-g9bfc0cf-1xenial ceph-test=10.2.2-508-g9bfc0cf-1xenial radosgw=10.2.2-508-g9bfc0cf-1xenial python-ceph=10.2.2-508-g9bfc0cf-1xenial libcephfs1=10.2.2-508-g9bfc0cf-1xenial libcephfs-java=10.2.2-508-g9bfc0cf-1xenial libcephfs-jni=10.2.2-508-g9bfc0cf-1xenial librados2=10.2.2-508-g9bfc0cf-1xenial librbd1=10.2.2-508-g9bfc0cf-1xenial rbd-fuse=10.2.2-508-g9bfc0cf-1xenial'
2016-09-09T14:55:07.483 INFO:teuthology.orchestra.run.vpm107.stdout:Reading package lists...
2016-09-09T14:55:07.644 INFO:teuthology.orchestra.run.vpm107.stdout:Building dependency tree...
2016-09-09T14:55:07.645 INFO:teuthology.orchestra.run.vpm107.stdout:Reading state information...
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'ceph' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'ceph-mds' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'ceph-common' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'ceph-fuse' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'ceph-test' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'radosgw' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'python-ceph' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'libcephfs1' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'libcephfs-java' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'libcephfs-jni' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'librados2' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'librbd1' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'rbd-fuse' was not found
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/parallel.py", line 83, in __exit__
  File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/parallel.py", line 101, in next
  File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/parallel.py", line 19, in capture_traceback
  File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/task/install.py", line 107, in _update_deb_package_list_and_install
  File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/orchestra/remote.py", line 192, in run
  File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/orchestra/run.py", line 399, in run
  File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/orchestra/run.py", line 166, in wait
CommandFailedError: Command failed on vpm107 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=10.2.2-508-g9bfc0cf-1xenial ceph-mds=10.2.2-508-g9bfc0cf-1xenial ceph-common=10.2.2-508-g9bfc0cf-1xenial ceph-fuse=10.2.2-508-g9bfc0cf-1xenial ceph-test=10.2.2-508-g9bfc0cf-1xenial radosgw=10.2.2-508-g9bfc0cf-1xenial python-ceph=10.2.2-508-g9bfc0cf-1xenial libcephfs1=10.2.2-508-g9bfc0cf-1xenial libcephfs-java=10.2.2-508-g9bfc0cf-1xenial libcephfs-jni=10.2.2-508-g9bfc0cf-1xenial librados2=10.2.2-508-g9bfc0cf-1xenial librbd1=10.2.2-508-g9bfc0cf-1xenial rbd-fuse=10.2.2-508-g9bfc0cf-1xenial'

adding new distro versions for DEB repos doesn't recreate the file

The code will only create the file if it doesn't exist, no way to force it to create one if modifying the templating.

update to python3 - take care of sqlalchemy CVE

The CVE is easy, but the python3 work is going to take a bit of effort. Simply bumping Sqlalchemy to 1.3.0 fixes both CVEs.

Update everything to work with Ubuntu 19.04

regardless if the repo is queued force it to be re-queued

Clients like chacractl will almost always want to forcefully try to recreate/create a repository, so the is_queued flag needs to be reset to False so that the celerybeat process will pick it up

automatic (configurable) time-to-deletion

From @alfredodeza on May 24, 2016 19:3

Binaries are huge, and for testing purposes there is no need to keep every binary and every repository built.

A configurable value (that defaults to two weeks) should be allowed to destroy a sha1/refs for a given project.

Copied from original issue: ceph/mita#75

cleanup the async modules, separating them logically

Currently there are tasks for creating deb and rpm repos and for polling but they all exist in async/__init__.py and it is becoming impossibly large.

These tasks should get separated, cleaned up, and use the proper Celery helpers to get them initialized and configured properly.

The health check endpoint should check disk usage

If shaman gives out a chacra node that has little disk space builds fail when trying to upload the binaries. We need some way to remove chacra nodes from the pool that are not capable of storing anymore binaries or repos.

At some point when those repos are auto deleted we'll need a mechanism for adding them back to the pool as well.

arm64 is not a supported architecture

Just armhf. So the template for DEB distributions needs to be updated.

binary create fails with HTTP 405 -- what am I missing?

(chacra)vagrant@vagrant:~$ pip freeze --version chacractl
argparse==1.2.1
chacractl==0.0.8
requests==2.10.0
tambo==0.2.0
wsgiref==0.1.2
(chacra)vagrant@vagrant:~$ chacra/bin/chacractl binary create calamari/1.4.7/centos/7/x86_64/ calamari-server-1.4.7.rpm 
[requests.packages.urllib3.connectionpool][INFO  ] Starting new HTTPS connection (1): chacra.ceph.com
/home/vagrant/chacra/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/home/vagrant/chacra/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
[requests.packages.urllib3.connectionpool][DEBUG ] "HEAD /binaries/calamari/1.4.7/centos/7/x86_64/calamari-server-1.4.7.rpm/ HTTP/1.1" 404 0
[chacractl.api.binaries][INFO  ] POSTing file: /home/vagrant/calamari-server-1.4.7.rpm
[requests.packages.urllib3.connectionpool][INFO  ] Starting new HTTPS connection (1): chacra.ceph.com
/home/vagrant/chacra/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
[requests.packages.urllib3.connectionpool][DEBUG ] "POST /binaries/calamari/1.4.7/centos/7/x86_64/ HTTP/1.1" 405 56
[chacractl.api.binaries][WARNIN] 405 -> {"message": "POST requests to this url are not allowed"}
Traceback (most recent call last):
  File "chacra/bin/chacractl", line 6, in <module>
    main.ChacraCtl()
  File "/home/vagrant/chacra/local/lib/python2.7/site-packages/chacractl/main.py", line 38, in __init__
    self.main(argv)
  File "/home/vagrant/chacra/local/lib/python2.7/site-packages/chacractl/decorators.py", line 68, in newfunc
    return f(*a, **kw)
  File "/home/vagrant/chacra/local/lib/python2.7/site-packages/chacractl/main.py", line 82, in main
    parser.dispatch()
  File "/home/vagrant/chacra/local/lib/python2.7/site-packages/tambo/dispatcher.py", line 21, in dispatch
    return instance.main()
  File "/home/vagrant/chacra/local/lib/python2.7/site-packages/chacractl/api/binaries.py", line 136, in main
    self.post(url, filepath)
  File "/home/vagrant/chacra/local/lib/python2.7/site-packages/chacractl/api/binaries.py", line 82, in post
    response.raise_for_status()
  File "/home/vagrant/chacra/local/lib/python2.7/site-packages/requests/models.py", line 844, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 405 Client Error: Method Not Allowed for url: https://chacra.ceph.com/binaries/calamari/1.4.7/centos/7/x86_64/
(chacra)vagrant@vagrant:~$

`ref` should be able to understand what `sha1` has it built

From @alfredodeza on May 24, 2016 18:55

This presents a problem because the current URL structure is:

 project/ref/distro/distro_version/architecture/

So modifying this to accommodate sha1 (backwards incompatible) would look like

 project/ref/sha1/distro/distro_version/architecture/

The other option would be to not make sha1 part of the url, but part of the API. That is, ref would need to be told about the sha1 it is receiving, but this sounds problematic as the correctness would fully rely on the client and not the system. It would be tricky to ensure separation of sha1's across the board.

For example, when building a repository, code would need to be adapted to not mix different binaries from different sha1's.

I am not sure what is the best answer here, some users may not even want to have a sha1 in the url and that would be cumbersome to make it optional.

Copied from original issue: ceph/mita#73

Add support for a generic blob repo

We can now build Windows binaries that are currently distributed via a zip file. Since this extension doesn't map to an RPM nor DEB repo, there is no easy way to access the artifacts via chacra repo url.

ceph / chacra Goto Github PK

chacra's People

Contributors

Stargazers

Watchers

Forkers

chacra's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs