unhandles TypeError when OutpuGood tests fail about autotest-docker

[cevich]

What version of docker autotest is this? What version of autotest (run client/shared/version.py)? I think this bug may have already been fixed, (I'll try again also to be sure). Thanks!

[cevich]

I tried reproducing this on several older versions of Docker Autotest (with latest autotest) but am not able 😦 For me, the run_twice test fails (as expected) when the error checker encounters 'error' on stderr. Here are all the details, please let me know if you can reproduce this with same/similar versions or if you can spot what the differences are:

[root@docker docker]# git reset --hard 0.5.5
HEAD is now at 833331b Docker: Release Update 0.5.5 (no API changes)
[root@docker docker]# ../../shared/version.py 
0.16.0-master-63-g35e93
[root@docker docker]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.0 (Maipo)
[root@docker docker]# rpm -q docker-io
docker-io-0.9.1-3.collider.el7.x86_64
[root@docker docker]# git am /tmp/63.patch 
Applying: subtests..run_twice: Add run container twice test Signed-off-by: Ouyang Guohua <[email protected]>
[root@docker docker]# cd ../..
[root@docker client]# reset; tools/make_clean; ./autotest-local run docker --verbose --args=docker_cli/run_twice
Cleaning aio_dio_bugs test dir
Cleaning cyclictest test dir
...
Cleaning tracing_microbenchmark test dir
Cleaning tsc test dir
09:32:37 INFO | Writing results to /usr/local/autotest/client/results/default
09:32:37 DEBUG| Initializing the state engine
09:32:37 DEBUG| Persistent state client.steps now set to []
09:32:37 DEBUG| Persistent option harness now set to None
09:32:37 DEBUG| Persistent option harness_args now set to None
09:32:37 DEBUG| Selected harness: standalone
09:32:38 INFO | START   ----    ----    timestamp=1397741558    localtime=Apr 17 09:32:38   
...
09:32:40 INFO |     START   docker/subtests/docker_cli/run_twice.test_1-of-1    docker/subtests/docker_cli/run_twice.test_1-of-1    timestamp=1397741560    timeout=600 localtime=Apr 17 09:32:40   
...
09:32:44 INFO |         RUNNING ----    ----    timestamp=1397741564    localtime=Apr 17 09:32:44   INFO: postprocess()
09:32:44 INFO |         RUNNING ----    ----    timestamp=1397741564    localtime=Apr 17 09:32:44   INFO: command: '/usr/bin/docker -D run --name=y1C5Ws6JB1t9 10.16.71.105:5000/fedora /bin/bash -c 'echo test''
09:32:44 INFO |         RUNNING ----    ----    timestamp=1397741564    localtime=Apr 17 09:32:44   INFO: command: '/usr/bin/docker -D run --name=y1C5Ws6JB1t9 10.16.71.105:5000/fedora /bin/bash -c 'echo test''
09:32:44 ERROR| Exception escaping from test:
Traceback (most recent call last):
  File "/usr/local/autotest/client/shared/test.py", line 411, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/client/shared/test.py", line 823, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/client/tests/docker/dockertest/subtest.py", line 117, in execute
    *args, **dargs)
  File "/usr/local/autotest/client/shared/test.py", line 298, in execute
    self.postprocess()
  File "/usr/local/autotest/client/tests/docker/subtests/docker_cli/run_twice/run_twice.py", line 53, in postprocess
    self.failif(not outputgood, str(outputgood))
  File "/usr/local/autotest/client/tests/docker/dockertest/subtest.py", line 171, in failif
    raise DockerTestFail(reason)
DockerTestFail: Good: ['crash_check_stdout', 'error_check_stdout', 'usage_check_stdout', 'crash_check_stderr', 'usage_check_stderr']; Not Good: ['error_check_stderr']; Details: (error_check_stderr, No details)
09:32:44 INFO |         RUNNING ----    ----    timestamp=1397741564    localtime=Apr 17 09:32:44   INFO: cleanup()
09:32:44 ERROR| child process failed
09:32:44 DEBUG| Traceback (most recent call last):
09:32:44 DEBUG|   File "/usr/local/autotest/client/parallel.py", line 25, in fork_start
09:32:44 DEBUG|     l()
09:32:44 DEBUG|   File "/usr/local/autotest/client/job.py", line 505, in <lambda>
09:32:44 DEBUG|     l = lambda: test.runtest(self, url, tag, args, dargs)
09:32:44 DEBUG|   File "/usr/local/autotest/client/test.py", line 125, in runtest
09:32:44 DEBUG|     job.sysinfo.log_after_each_iteration)
09:32:44 DEBUG|   File "/usr/local/autotest/client/shared/test.py", line 913, in runtest
09:32:44 DEBUG|     mytest._exec(args, dargs)
09:32:44 DEBUG|   File "/usr/local/autotest/client/shared/test.py", line 411, in _exec
09:32:44 DEBUG|     _call_test_function(self.execute, *p_args, **p_dargs)
09:32:44 DEBUG|   File "/usr/local/autotest/client/shared/test.py", line 823, in _call_test_function
09:32:44 DEBUG|     return func(*args, **dargs)
09:32:44 DEBUG|   File "/usr/local/autotest/client/tests/docker/dockertest/subtest.py", line 117, in execute
09:32:44 DEBUG|     *args, **dargs)
09:32:44 DEBUG|   File "/usr/local/autotest/client/shared/test.py", line 298, in execute
09:32:44 DEBUG|     self.postprocess()
09:32:44 DEBUG|   File "/usr/local/autotest/client/tests/docker/subtests/docker_cli/run_twice/run_twice.py", line 53, in postprocess
09:32:44 DEBUG|     self.failif(not outputgood, str(outputgood))
09:32:44 DEBUG|   File "/usr/local/autotest/client/tests/docker/dockertest/subtest.py", line 171, in failif
09:32:44 DEBUG|     raise DockerTestFail(reason)
09:32:44 DEBUG| DockerTestFail: Good: ['crash_check_stdout', 'error_check_stdout', 'usage_check_stdout', 'crash_check_stderr', 'usage_check_stderr']; Not Good: ['error_check_stderr']; Details: (error_check_stderr, No details)
09:32:44 INFO |         FAIL    docker/subtests/docker_cli/run_twice.test_1-of-1    docker/subtests/docker_cli/run_twice.test_1-of-1    timestamp=1397741564    localtime=Apr 17 09:32:44   Good: ['crash_check_stdout', 'error_check_stdout', 'usage_check_stdout', 'crash_check_stderr', 'usage_check_stderr']; Not Good: ['error_check_stderr']; Details: (error_check_stderr, No details)
09:32:45 INFO |     END FAIL    docker/subtests/docker_cli/run_twice.test_1-of-1    docker/subtests/docker_cli/run_twice.test_1-of-1    timestamp=1397741565    localtime=Apr 17 09:32:45   
09:32:45 DEBUG| Persistent state client._record_indent now set to 1
09:32:45 DEBUG| Persistent state client.unexpected_reboot deleted
09:32:45 DEBUG| Persistent state client.steps now set to []
09:32:45 INFO | END GOOD    ----    ----    timestamp=1397741565    localtime=Apr 17 09:32:45   
09:32:45 DEBUG| Persistent state client._record_indent now set to 0
09:32:45 INFO | Report successfully generated at /usr/local/autotest/client/results/default/job_report.html

[gouyang]

I meet the issue on RHEL65 when test both run_twice and dockerhelp.

[root@rhevm-si-11-225 docker]# ../../shared/version.py
0.16.0-master-83-g4e1a
[root@rhevm-si-11-225 docker]# rpm -q docker-io
docker-io-0.9.0-3.el6.x86_64
[root@rhevm-si-11-225 docker]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.5 (Santiago)

[root@rhevm-si-11-225 tests]# rm -rf docker/
[root@rhevm-si-11-225 tests]# git clone https://github.com/autotest/autotest-docker.git docker
Initialized empty Git repository in /var/www/html/autotest/client/tests/docker/.git/
remote: Reusing existing pack: 877, done.
remote: Counting objects: 89, done.
remote: Compressing objects: 100% (82/82), done.
remote: Total 966 (delta 31), reused 0 (delta 0)
Receiving objects: 100% (966/966), 302.10 KiB | 181 KiB/s, done.
Resolving deltas: 100% (493/493), done.
[root@rhevm-si-11-225 tests]# cd docker/
[root@rhevm-si-11-225 docker]# mkdir config_custom
[root@rhevm-si-11-225 docker]# cp -rp config_defaults/defaults.ini config_custom/defaults.ini
[root@rhevm-si-11-225 docker]# vi config_custom/defaults.ini
[root@rhevm-si-11-225 docker]# sed -i 's#True#False#' subtests/docker_cli/dockerhelp/dockerhelp.py
[root@rhevm-si-11-225 docker]# ../../autotest-local run docker --args=docker_cli/dockerhelp
22:37:57 INFO | Writing results to /var/www/html/autotest/client/results/default
22:37:57 INFO | START ---- ---- timestamp=1397745477 localtime=Apr 17 22:37:57
22:37:57 INFO | Good: []; Not Good: ['non-default-images.sh']; Details: (non-default-images.sh, {'exit': 3, 'stderr': 'Found unexpected image: test_4izFnv5k latest\ntest_77nAmElZ latest\ntest_Hfemvxlg latest\ntest_Hfemvxlg latest_daVmmVil\ntest_MsAiMpGU latest\ntest_gWASCNIK latest\ntest_CIlgutCe latest\ntest_KjU2Y801 latest\ntest_hVmLk4pl latest\ntest_hVmLk4pl latest_QhU8G5zE\ntest_JGXhlsx1 latest\ntest_KYEspwg9 latest\ntest_Oa4oPBtA/test_gWASCNIK latest\nfedora 20\nfedora latest\ntest_5enB1ztA latest\ntest_TQebyyg1 latest\ntest_W9gZNeSB/test_KYEspwg9 latest 105182bb5e8bed742e4063e9486afa5f51328fc007e74f6c9f29cd1f75971357\n', 'stdout': ''})
22:37:57 INFO | Environment checks failed! Blame BOFH
22:37:57 INFO | START docker/subtests/docker_cli/dockerhelp.test_1-of-1 docker/subtests/docker_cli/dockerhelp.test_1-of-1 timestamp=1397745477 timeout=600 localtime=Apr 17 22:37:57
22:37:58 INFO | RUNNING ---- ---- timestamp=1397745478 localtime=Apr 17 22:37:58 INFO: initialize()
22:37:58 INFO | RUNNING ---- ---- timestamp=1397745478 localtime=Apr 17 22:37:58 INFO: setup() for subtest version 1285
22:37:58 INFO | RUNNING ---- ---- timestamp=1397745478 localtime=Apr 17 22:37:58 INFO: run_once() iteration 1 of 1
22:37:58 INFO | RUNNING ---- ---- timestamp=1397745478 localtime=Apr 17 22:37:58 INFO: postprocess_iteration(), iteration #1
22:37:58 INFO | RUNNING ---- ---- timestamp=1397745478 localtime=Apr 17 22:37:58 INFO: postprocess()
22:37:58 INFO | RUNNING ---- ---- timestamp=1397745478 localtime=Apr 17 22:37:58 INFO: command: '/usr/bin/docker -D help'
22:37:58 INFO | RUNNING ---- ---- timestamp=1397745478 localtime=Apr 17 22:37:58 INFO: command: '/usr/bin/docker -D foobar'
22:37:58 INFO | RUNNING ---- ---- timestamp=1397745478 localtime=Apr 17 22:37:58 INFO: command: '/usr/bin/docker -D " "'
22:37:58 INFO | RUNNING ---- ---- timestamp=1397745478 localtime=Apr 17 22:37:58 INFO: command: '/usr/bin/docker -D '''
22:37:58 ERROR| Exception escaping from test:
Traceback (most recent call last):
File "/var/www/html/autotest/client/shared/test.py", line 411, in _exec
_call_test_function(self.execute, _p_args, *_p_dargs)
File "/var/www/html/autotest/client/shared/test.py", line 823, in _call_test_function
return func(_args, *_dargs)
File "/var/www/html/autotest/client/tests/docker/dockertest/subtest.py", line 117, in execute
_args, *_dargs)
File "/var/www/html/autotest/client/shared/test.py", line 298, in execute
self.postprocess()
File "/var/www/html/autotest/client/tests/docker/subtests/docker_cli/dockerhelp/dockerhelp.py", line 62, in postprocess
skip=['usage_check'])
File "/var/www/html/autotest/client/tests/docker/dockertest/output.py", line 373, in init
raise xceptions.DockerOutputError(str(self))
DockerOutputError: Good: ['error_check_stdout', 'crash_check_stdout']; Not Good: ['error_check_stderr', 'crash_check_stderr']; Details: (error_check_stderr, No details); (crash_check_stderr, No details)
22:37:58 INFO | RUNNING ---- ---- timestamp=1397745478 localtime=Apr 17 22:37:58 INFO: cleanup()
22:37:58 ERROR| child process failed
22:37:59 INFO | ERROR docker/subtests/docker_cli/dockerhelp.test_1-of-1 docker/subtests/docker_cli/dockerhelp.test_1-of-1 timestamp=1397745479 localtime=Apr 17 22:37:59 Unhandled TypeError: init() takes exactly 2 arguments (1 given)
Traceback (most recent call last):
File "/var/www/html/autotest/client/job.py", line 510, in _runtest
parallel.fork_waitfor_timed(self.resultdir, pid, timeout)
File "/var/www/html/autotest/client/parallel.py", line 116, in fork_waitfor_timed
_check_for_subprocess_exception(tmp, pid)
File "/var/www/html/autotest/client/parallel.py", line 67, in _check_for_subprocess_exception
e = pickle.load(file(ename, 'r'))
File "/usr/lib64/python2.6/pickle.py", line 1370, in load
return Unpickler(file).load()
File "/usr/lib64/python2.6/pickle.py", line 858, in load
dispatchkey
File "/usr/lib64/python2.6/pickle.py", line 1133, in load_reduce
value = func(*args)
TypeError: init() takes exactly 2 arguments (1 given)

22:37:59 INFO | END ERROR docker/subtests/docker_cli/dockerhelp.test_1-of-1 docker/subtests/docker_cli/dockerhelp.test_1-of-1 timestamp=1397745479 localtime=Apr 17 22:37:59
22:37:59 INFO | Good: []; Not Good: ['non-default-images.sh']; Details: (non-default-images.sh, {'exit': 3, 'stderr': 'Found unexpected image: test_CIlgutCe latest\ntest_KjU2Y801 latest\ntest_hVmLk4pl latest\ntest_hVmLk4pl latest_QhU8G5zE\ntest_JGXhlsx1 latest\ntest_KYEspwg9 latest\ntest_Oa4oPBtA/test_gWASCNIK latest\nfedora 20\nfedora latest\ntest_5enB1ztA latest\ntest_TQebyyg1 latest\ntest_W9gZNeSB/test_KYEspwg9 latest\ntest_4izFnv5k latest\ntest_77nAmElZ latest\ntest_Hfemvxlg latest\ntest_Hfemvxlg latest_daVmmVil\ntest_MsAiMpGU latest\ntest_gWASCNIK latest 105182bb5e8bed742e4063e9486afa5f51328fc007e74f6c9f29cd1f75971357\n', 'stdout': ''})
22:37:59 INFO | Environment checks failed! Blame docker/subtests/docker_cli/dockerhelp
22:37:59 INFO | END GOOD ---- ---- timestamp=1397745479 localtime=Apr 17 22:37:59
22:37:59 INFO | Report successfully generated at /var/www/html/autotest/client/results/default/job_report.html

[gouyang]

you did not change ignore_error to False or remove it.

[cevich]

Thanks for the info, unf. I'm outta time for today, I'll take a look at this again on Monday. Though things seem to be working okay for me on 0.6.1 (about to merge it).

[cevich]

Ahh gotcha, raise xceptions.DockerOutputError(str(self)) is expected, it's the odd-ball autotest exception this issue is about.

File "/usr/lib64/python2.6/pickle.py", line 1133, in load_reduce
value = func(*args)
TypeError: __init() takes exactly 2 arguments (1 given)

Something must have changed recently in autotest to cause this behaviour, because I've seen it in other contexts as well. I'll use this reproducer and bisect to see if I can find where it's coming from.