cki-project / skt Goto Github PK

Make the code follow single naming conventions. E.g. from naming conventions from PEP8.

If a run command is interrupted after the Beaker job is submitted, but before it starts running (i.e. while it is still "queued"), the report command running afterwards aborts with the following in the (-vv) output:

2018-04-12 12:09:46,314     INFO   runner type: beaker
2018-04-12 12:09:46,314     INFO   beaker template: beakerjob.xml
Traceback (most recent call last):
  File "/home/worker/runner/workspace/rhel-7.5@2@tmp/virtualenv/bin/skt", line 11, in <module>
    load_entry_point('skt==1', 'console_scripts', 'skt')()
  File "/home/worker/runner/workspace/rhel-7.5@2@tmp/virtualenv/lib/python2.7/site-packages/skt/executable.py", line 710, in main
    args.func(cfg)
  File "/home/worker/runner/workspace/rhel-7.5@2@tmp/virtualenv/lib/python2.7/site-packages/skt/executable.py", line 332, in cmd_report
    reporter.report()
  File "/home/worker/runner/workspace/rhel-7.5@2@tmp/virtualenv/lib/python2.7/site-packages/skt/reporter.py", line 465, in report
    msg.attach(MIMEText(self.getreport()))
  File "/home/worker/runner/workspace/rhel-7.5@2@tmp/virtualenv/lib/python2.7/site-packages/skt/reporter.py", line 394, in getreport
    msg += self.getjobresults()
  File "/home/worker/runner/workspace/rhel-7.5@2@tmp/virtualenv/lib/python2.7/site-packages/skt/reporter.py", line 333, in getjobresults
    vresults = runner.getverboseresults(list(self.cfg.get("jobs")))
TypeError: 'NoneType' object is not iterable

Instead it should proceed and report appropriate results.

Consider using LinchPin to abstract away skt's dependence on Beaker for running tests. This might allow us to support running tests with libvirt to support a simple development setup. See also LinchPin documentation.

We have to carefully evaluate LinchPin, though. It might be simpler and more reliable to implement our own abstraction for just two providers: Beaker and libvirt, given our relatively minor feature requirements. LinchPin might also bring in too many dependencies.

Go over the code and either fix, or convert all the TODOs and FIXMEs in the code to issues, then remove them from the code.

Document all the code following the conventions for Google Python Style Guide.

To be able to send results to our result maillists without disclosing their addresses we need support for adding their addresses to BCC of report messages.

Cloning a repository with --depth <somenumber> is much faster than cloning the entire repo, especially when we want to quickly check if a patch merges. skt needs support for the --depth argument.

Once this is done, the Circle-CI jobs should be adjusted to use skt for the cloning instead (see #59).

While the ktree.merge_patchwork_patch provides logs about the patch application failure, the sister method ktree.merge_patch_file does not. This means we don't have any feedback to report to the patch author if the patch failed to apply.

Because of the similarity between the functions that would be introduced by the fix, it would be nice if we could make the merge_patchwork_patch retrieve the patch file and then use merge_patch_file internally.

At the moment, we replace all commas with semicolons in patch names (subjects) so that they do not clash with commas in CSV files we write later. Stop doing that by e.g. using a proper CSV writer (from the csv module), or better yet by switching to a structured file format which would also get rid of the structure depending on the order of class member calls.

Now that #22 has merged, pip should automatically install any required dependencies automatically when skt is installed.

Rename the skt.py program to just skt. We don't need the .py on the end, and we don't need it to be confused with a module.

The developer docs are missing instructions for installing skt into a user's home directory or in a virtual environment.

When a Beaker job is submitted, print its web UI URL in the logs. This way it would be easier to open it in a browser and see its progress when looking through or watching skt logs.

When skt submits a Beaker job, and the job is canceled while still queued, skt sends the following report message:

result report for kernel 3.10.0+

-----------------------
base repo: git://git.app.eng.bos.redhat.com/rhel7.git
     HEAD: 52a49df65d752a98fef13507f4682d50ba4d010c

config: see attached 'config.gz'

-----------------------
Tested:
  - Boot test

-----------------------
Test Run: #1
result: Warn
---

The skt -vv --state --junit junit --rc sktrc run --wait command produces the
following output when the job is cancelled:

2018-04-13 07:59:42,800     INFO   runner type: beaker
2018-04-13 07:59:42,800     INFO   beaker template: beakerjob.xml
2018-04-13 07:59:43,699     INFO   submitted jobid: J:2424443
2018-04-13 07:59:44,099     INFO   added R:5036489 to watchlist
2018-04-13 08:09:49,371     INFO   R:5036489 status changed to 'Cancelled', removing from watchlist
2018-04-13 08:09:49,734    DEBUG   state: jobid_0 -> J:2424443
2018-04-13 08:09:49,735    DEBUG   state: retcode -> 0

At present, invocations of skt without patches applied produce the following
in reports:

We applied the following patch:

on top of commit 23b389c23139 from the repository at
   git://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git 

Fix the reports to produce appropriate text for these cases.

We need a constraint that causes CI to use a working version of pycodestyle until the issue is fixed.

The README.md is getting fairly lengthy and it might be time for documentation written in Sphinx.

Skt produces an exception with barely-related traceback when it fails to submit a Beaker job:

+ skt -vv --state --rc skt-rc --junit junit run --wait
2018-04-18 13:17:58,346     INFO   runner type: beaker
2018-04-18 13:17:58,346     INFO   beaker template: beakerjob.xml

Configuration file not found. Please create an /etc/beaker/client.conf or ~/.beaker_client/config configuration file.
2018-04-18 13:17:58,769     INFO   submitted jobid: None
2018-04-18 13:17:58,776    ERROR   Exception caught: Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/sktm@tmp/virtualenv/lib/python2.7/site-packages/skt/executable.py", line 102, in wrapper
    func(cfg)
  File "/var/lib/jenkins/workspace/sktm@tmp/virtualenv/lib/python2.7/site-packages/skt/executable.py", line 281, in cmd_run
    cfg.get('wait'), uid=cfg.get('uid'))
  File "/var/lib/jenkins/workspace/sktm@tmp/virtualenv/lib/python2.7/site-packages/skt/runner.py", line 433, in run
    self.wait(jobid, reschedule)
  File "/var/lib/jenkins/workspace/sktm@tmp/virtualenv/lib/python2.7/site-packages/skt/runner.py", line 369, in wait
    self.add_to_watchlist(jobid, reschedule)
  File "/var/lib/jenkins/workspace/sktm@tmp/virtualenv/lib/python2.7/site-packages/skt/runner.py", line 351, in add_to_watchlist
    root = self.getresultstree(jobid)
  File "/var/lib/jenkins/workspace/sktm@tmp/virtualenv/lib/python2.7/site-packages/skt/runner.py", line 115, in getresultstree
    bkr = subprocess.Popen(args, stdout=subprocess.PIPE)
  File "/usr/lib64/python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1025, in _execute_child
    raise child_exception
TypeError: execv() arg 2 must contain only strings

Skt should handle the error, abort immediately, and produce a clear error message.

The main idea is that the CI pipeline takes care the new code being lint compliant. For now, it can be run only for the tests folder but judging for the issue #3 all the code are going to be lint compliant 😄

As of PR #59, Circle-CI clones the master branch of the kernel tree and applies a patch. If the patch stops applying cleanly, our CI jobs will break. We need to use a stable kernel source branch for this CI job.

Reproduce: just run skt --state --wipe cleanup (no steps before, don't have any rc or workdir)

Traceback (most recent call last):
  File "/usr/bin/skt", line 11, in <module>
    load_entry_point('skt', 'console_scripts', 'skt')()
  File "/home/vkabatova/git/skt/skt/executable.py", line 769, in main
    args.func(cfg)
  File "/home/vkabatova/git/skt/skt/executable.py", line 312, in cmd_cleanup
    shutil.rmtree(os.path.expanduser(cfg.get('workdir')))
  File "/usr/lib64/python2.7/posixpath.py", line 254, in expanduser
    if not path.startswith('~'):
AttributeError: 'NoneType' object has no attribute 'startswith'

This should be easily solvable by checking if the workdir is specified before attempting to remove it (specifying a default to .get() won't help, just change the error to nonexistent file).

At the moment skt workflow state is stored in the configuration file, which makes its reuse difficult and error-prone. Move the state somewhere else.

I got error when I try to install skt with the beaker extras:

$ pip install git+https://github.com/RH-FMK/skt[beaker]

The error log:

Collecting git+https://github.com/RH-FMK/skt[beaker]
  Cloning https://github.com/RH-FMK/skt[beaker] to /tmp/pip-req-build-d_wNVQ
fatal: unable to access 'https://github.com/RH-FMK/skt[beaker]/': The requested URL returned error: 400
Command "git clone -q https://github.com/RH-FMK/skt[beaker] /tmp/pip-req-build-d_wNVQ" failed with error code 128 in None

Could you help me figure out it? Thanks

We need to document how skt supplies parameters to the kernel build system, which parameters, and what it expects in return. E.g. which targets it invokes, which variables it sets, how it supplies the configuration, how it finds the resulting tarballs, etc. This would allow users and developers to understand how skt works and what is and what is not important, how passing arbitrary arguments to make through skt would affect the build, and finally this would allow us to implement a tool for generating mock-up kernel sources, which checkout and build much faster than the real kernel, satisfy skt, and allow us to speedup debugging and testing dramatically.

Possibly put the jenkins build ID into beaker job whiteboard?

We should run flake8 against the code in Travis to ensure we're following good coding guidelines. Linting should also happen before the code is tested to save time.

see ktree.merge_patchwork_patch . This can be an issue especially with Patchwork2 instances that have the legacy interface turned off. We need to fix the patch retrieval, either using direct mbox URLs or a wrapper that would check which interface (REST/XMLRPC) is available and use that.

Hello developers,
The skt project is really interesting. I followes the introduction from README to install and try some command option, but I am not sure why I got the error when I try to fetch a kernel repository following merge command. In particular, I try to run this command to checkout master branch of the net-next repo:

$ skt --rc skt-rc --state --workdir skt-workdir -vv \
         merge --baserepo git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git \
               --ref master

And then I got the error:

2018-04-20 09:05:25,164    DEBUG   executing: git --work-tree skt-workdir --git-dir skt-workdir/.git fetch -n origin +master:refs/remotes/origin/master
fatal: unable to connect to git.kernel.org:
git.kernel.org[0: 147.75.58.133]: errno=Connection refused
git.kernel.org[1: 2604:1380:4020:600::1]: errno=Network is unreachable

Traceback (most recent call last):
  File "/usr/bin/skt", line 11, in <module>
    load_entry_point('skt', 'console_scripts', 'skt')()
  File "/home/danghai/skt/skt/executable.py", line 710, in main
    args.func(cfg)
  File "/home/danghai/skt/skt/executable.py", line 116, in wrapper
    func(cfg)
  File "/home/danghai/skt/skt/executable.py", line 137, in cmd_merge
    bhead = ktree.checkout()
  File "/home/danghai/skt/skt/__init__.py", line 291, in checkout
    self.git_cmd(*git_fetch_args)
  File "/home/danghai/skt/skt/__init__.py", line 189, in git_cmd
    subprocess.check_call(args, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', '--work-tree', 'skt-workdir', '--git-dir', 'skt-workdir/.git', 'fetch', '-n', 'origin', '+master:refs/remotes/origin/master']' returned non-zero exit status 128

However, when I change the URL: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git, it is able to fetch. But it is frozen after completely fetch. I must terminate it . It shows:

2018-04-20 09:00:40,404    DEBUG   executing: git --work-tree skt-work --git-dir skt-work/.git fetch -n origin +master:refs/remotes/origin/master
remote: Counting objects: 5995556, done.
remote: Compressing objects: 100% (910905/910905), done.
remote: Total 5995556 (delta 5042858), reused 5992915 (delta 5041111)
Receiving objects: 100% (5995556/5995556), 1.01 GiB | 2.09 MiB/s, done.
Resolving deltas: 100% (5042858/5042858), done.

Could you help me handle these error? Thanks

The debuginfo option is enabled by default and we need an option to disable it. For merge/build/boot tests, those debug symbols aren't critical.

A kernel built with debuginfo is ~ 250MB (tar.gz) and one without is ~ 70MB.

The list of parsers in executable.py are difficult to read and could be improved with proper indentation.

I'll take a crack at this.

When using git --work-tree=... --git-dir=... am - on CentOS 7, you get an error:

fatal: /usr/libexec/git-core/git-am cannot be used without a working tree.

The git version in CentOS 7 is too old for this to work properly and it's been fixed in later versions. We have a few options:

• Use GitPython in skt (probably a good idea)
• chdir into the work directory to apply patches (would make paths confusing)
• Upgrade git whenever we use CentOS 7 (sounds awful)

Document a functional setup with configuration suitable for testing.

After executing the following command right outside the workdir (skt-workdir):

skt --rc skt-rc --state --workdir skt-workdir -vv \
    merge --baserepo git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git \
          --ref a870a02cc963de35452bbed932560ed69725c4f2 \
          --pw https://patchwork.ozlabs.org/patch/886637

The workdir has skt-workdir directory created under it.

When running skt in a container, it would be helpful to build kernel artifacts outside the workdir where the kernel source is cloned. This would allow for artifacts to be sent to other systems.

When a user runs skt, the path expansion should happen as early as possible. This expansion includes expanding home directory paths (~/some_dir/some_file.txt) or relative paths into absolute paths. This prevents situations where some parts of skt might handle paths in different ways than other parts.

If a user specifies jobtemplate.xml as the path to their Beaker XML template, we should look for that in the current working directory.

In Python3, print statement was promoted to print() function. As one of first steps for compatibility, move to using print() when needed (don't forget to import print_function from __future__ module).

At the moment we install skt along with its dependencies for every job, which periodically leads to breakage due to PyPI availability and package dependency issues. Find a way to isolate us at least from dependency issues, and preferably from availability issues as well.

The (non-verbose) kernel build log can be quite large, around 500K, so consider compressing it when sending the report messages.

To avoid exposing too much interface, prepend all private class method names with __ to have them recognized as such.

When skt is installed with pip inside a virtualenv, its execution of bkr command fails with something like this:

Traceback (most recent call last):
  File "/home/worker/runner/workspace/skt_test@tmp/virtualenv/bin/bkr", line 7, in <module>
    from bkr.client.main import main
  File "/home/worker/runner/workspace/skt_test@tmp/virtualenv/lib/python2.7/site-packages/bkr/client/main.py", line 16, in <module>
    import krbV

XMLRPC leads to a lot of headaches and requires lots of wrapper code. All of the data we need about a particular patch is held within the mbox file of the patch itself. We should just download the mbox file and parse it.

Running skt publish without --tarpkg produces stacktrace:

Traceback (most recent call last):
  File "skt/skt.py", line 569, in <module>
    main()
  File "skt/skt.py", line 559, in main
    args.func(cfg)
  File "skt/skt.py", line 99, in wrapper
    func(cfg)
  File "skt/skt.py", line 202, in cmd_publish
    url = publisher.publish(cfg.get('tarpkg'))
  File "/home/nkondras/skt/skt/publisher.py", line 60, in publish
    shutil.copy(os.path.expanduser(source), self.destination)
  File "/usr/lib64/python2.7/posixpath.py", line 254, in expanduser
    if not path.startswith('~'):
AttributeError: 'NoneType' object has no attribute 'startswith'

A readable error message about missing option should be produced instead.

The arguments list for skt.kbuilder is getting a bit long with the addition of enable_debuginfo. We should change this to use named arguments instead.

This was noted by @spbnick in #33.

At the moment, skt gets a count of CPUs on the system and uses that amount for kernel builds. However, there are some situations where this number needs to be increased or decreased. Users should be able to override this variable if needed.

Make the code comply with Google Python Style Guide, where possible. Where not possible, document exceptions or additions. Document our compliance and requirements for contributions. Proceed gradually. Add pylint checking to CI runs.