foundriesio / conductor

Test scheduler and coordinator. Works together with jobserv and LAVA

Setup rabbitmq for the worker

sudo rabbitmqctl add_vhost conductor
sudo rabbitmqctl add_user "conductor" "secret"
sudo rabbitmqctl set_permissions -p "conductor" "conductor" ".*" ".*" ".*"

Start worker

python -m celery -A conductor worker -B

conductor's Issues

commit not found in lmp-manifest when pushing to meta-sub

worker_1     | [2021-11-08 18:15:04,266: DEBUG/ForkPoolWorker-3] Starting new HTTPS connection (1): api.foundries.io:443
worker_1     | [2021-11-08 18:15:04,775: DEBUG/ForkPoolWorker-3] https://api.foundries.io:443 "GET /projects/lmp-ci-testing-next/lmp/builds/35/runs/raspberrypi4-64/.rundef.json HTTP/1.1" 200 None
worker_1     | [2021-11-08 18:15:04,783: DEBUG/ForkPoolWorker-3] Popen(['git', 'pull', '-v', 'origin', 'master'], cwd=/app/repositories/lmp-ci-testing-next, universal_newlines=True, shell=None, istream=None)
worker_1     | [2021-11-08 18:15:06,477: DEBUG/ForkPoolWorker-3] Fetch head lines do not match lines provided via progress information
worker_1     | length of progress lines 2 should be equal to lines in FETCH_HEAD file 1
worker_1     | Will ignore extra progress lines or fetch head lines.
worker_1     | [2021-11-08 18:15:06,478: DEBUG/ForkPoolWorker-3] b"info lines: [' * branch            master     -> FETCH_HEAD', '   d9e0a83..332905c  master     -> origin/master']"
worker_1     | [2021-11-08 18:15:06,478: DEBUG/ForkPoolWorker-3] b'head info: ["332905cda92e4ea212c6913a03ad622f0972cd85\\t\\tbranch \'master\' of https://source.foundries.io/factories//lmp-ci-testing-next/lmp-manifest\\n"]'
worker_1     | [2021-11-08 18:15:06,478: DEBUG/ForkPoolWorker-3] Popen(['git', 'cat-file', '--batch-check'], cwd=/app/repositories/lmp-ci-testing-next, universal_newlines=False, shell=None, istream=<valid stream>)
worker_1     | [2021-11-08 18:15:06,499: ERROR/ForkPoolWorker-3] Task conductor.core.tasks.update_build_commit_id[ecd5b684-bc4a-42ac-abfd-1f8c091d0b9d] raised unexpected: ValueError("SHA b'c34f7d9334970abeea4b18978608f6cda4f01893' could not be resolved, git returned: b'c34f7d9334970abeea4b18978608f6cda4f01893 missing'")
worker_1     | Traceback (most recent call last):
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 450, in trace_task
worker_1     |     R = retval = fun(*args, **kwargs)
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 731, in __protected_call__
worker_1     |     return self.run(*args, **kwargs)
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/conductor-1.24.3-py3.8.egg/conductor/core/tasks.py", line 261, in update_build_commit_id
worker_1     |     _update_build_reason(build)
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/conductor-1.24.3-py3.8.egg/conductor/core/tasks.py", line 229, in _update_build_reason
worker_1     |     commit = repository.commit(rev=build.commit_id)
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/git/repo/base.py", line 536, in commit
worker_1     |     return self.rev_parse(str(rev) + "^0")
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/git/repo/fun.py", line 229, in rev_parse
worker_1     |     obj = cast(Commit_ish, name_to_object(repo, rev[:start]))
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/git/repo/fun.py", line 166, in name_to_object
worker_1     |     return Object.new_from_sha(repo, hex_to_bin(hexsha))
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/git/objects/base.py", line 85, in new_from_sha
worker_1     |     oinfo = repo.odb.info(sha1)
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/git/db.py", line 43, in info
worker_1     |     hexsha, typename, size = self._git.get_object_header(bin_to_hex(binsha))
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/git/cmd.py", line 1252, in get_object_header
worker_1     |     return self.__get_object_header(cmd, ref)
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/git/cmd.py", line 1239, in __get_object_header
worker_1     |     return self._parse_object_header(cmd.stdout.readline())
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/git/cmd.py", line 1199, in _parse_object_header
worker_1     |     raise ValueError("SHA %s could not be resolved, git returned: %r" % (tokens[0], header_line.strip()))
worker_1     | ValueError: SHA b'c34f7d9334970abeea4b18978608f6cda4f01893' could not be resolved, git returned: b'c34f7d9334970abeea4b18978608f6cda4f01893 missing'

Error fetching results from LAVA

worker_1     | [2021-11-19 11:46:19,104: ERROR/ForkPoolWorker-3] Task conductor.core.tasks.process_testjob_notification[511fc217-ba7b-4053-b17e-8428aa0c1f99] raised unexpected: KeyError('definitions')
worker_1     | Traceback (most recent call last):
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 450, in trace_task
worker_1     |     R = retval = fun(*args, **kwargs)
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 731, in __protected_call__
worker_1     |     return self.run(*args, **kwargs)
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/conductor-1.24.6-py3.8.egg/conductor/core/tasks.py", line 554, in process_testjob_notification
worker_1     |     retrieve_lava_results(lava_db_device.id, job_id)
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/celery/local.py", line 188, in __call__
worker_1     |     return self._get_current_object()(*a, **kw)
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 732, in __protected_call__
worker_1     |     return orig(self, *args, **kwargs)
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/celery/app/task.py", line 389, in __call__
worker_1     |     return self.run(*args, **kwargs)
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/conductor-1.24.6-py3.8.egg/conductor/core/tasks.py", line 511, in retrieve_lava_results
worker_1     |     lava_results = __get_testjob_results__(lava_db_device, job_id)
worker_1     |   File "/usr/local/lib/python3.8/dist-packages/conductor-1.24.6-py3.8.egg/conductor/core/tasks.py", line 438, in __get_testjob_results__
worker_1     |     for expected_test in action['test']['definitions']:
worker_1     | KeyError: 'definitions'