Failure on sefttest about tbot

[Rahix]

Hi!

Hmm, this means that there is still some child process running ... Can you try applying the following patch?

diff --git a/tbot/machine/channel/subprocess.py b/tbot/machine/channel/subprocess.py
index 75d1846bb11f..78ac9297a5ba 100644
--- a/tbot/machine/channel/subprocess.py
+++ b/tbot/machine/channel/subprocess.py
@@ -94,6 +94,13 @@ class SubprocessChannelIO(channel.ChannelIO):
                 != 0
             ):
                 break
+
+            ret = subprocess.run(
+                ["ps", "-s", str(sid), "-o", "uid,pid,command"], capture_output=True
+            )
+            __import__("tbot").log.warning(
+                "Some child processes are still running:\n" + ret.stdout.decode()
+            )
             time.sleep(2 ** t / 100)
         else:
             raise Exception("not done")

It should then show the offending child processes whenever the error case you saw happens. I might be able to narrow down the reason with that info.

[sjg20]

Yes, here you go. Also, is there a quick way to run the local tbot rather than installing it each time?

tbot selftest
tbot starting ...
├─Calling selftest ...
│ ├─Calling testsuite ...
│ │ ├─Calling selftest_failing ...
│ │ │ ├─Calling inner ...
│ │ │ │ └─Fail. (0.000s)
│ │ │ └─Done. (0.001s)
│ │ ├─Calling selftest_skipping ...
│ │ │ ├─Calling inner ...
│ │ │ │ └─Skipped: This test is skipped on purpose
│ │ │ └─Done. (0.001s)
│ │ ├─Calling selftest_uname ...
│ │ │ └─Done. (0.003s)
│ │ ├─Calling selftest_user ...
│ │ │ └─Done. (0.002s)
│ │ ├─Calling selftest_machine_reentrant ...
│ │ │ └─Done. (0.001s)
│ │ ├─Calling selftest_machine_labhost_shell ...
│ │ │ ├─Calling selftest_machine_shell ...
│ │ │ │ ├─Testing command output ...
│ │ │ │ ├─Testing return codes ...
│ │ │ │ ├─Testing env vars ...
│ │ │ │ ├─Testing redirection (and weird paths) ...
│ │ │ │ ├─Testing subshell ...
│ │ │ │ └─Done. (0.053s)
│ │ │ ├─Calling selftest_machine_channel ...
│ │ │ │ └─Skipped: Channel tests need to be reimplemented for machine-v2
│ │ │ ├─Calling selftest_machine_channel ...
│ │ │ │ └─Skipped: Channel tests need to be reimplemented for machine-v2
│ │ │ └─Done. (0.149s)
│ │ ├─Calling selftest_machine_ssh_shell ...
│ │ │ ├─Calling check_minisshd ...
│ │ │ │ └─Done. (0.006s)
│ │ │ └─Skipped: dropbear is not installed so ssh can't be tested
│ │ ├─Calling selftest_machine_sshlab_shell ...
│ │ │ ├─Calling check_minisshd ...
│ │ │ │ └─Done. (0.005s)
│ │ │ └─Skipped: dropbear is not installed so ssh can't be tested
│ │ ├─Calling selftest_path_stat ...
│ │ │ ├─Setting up test files ...
│ │ │ ├─Checking existence ...
│ │ │ ├─Checking file modes ...
│ │ │ ├─Checking file modes on nonexistent files ...
│ │ │ ├─Checking stat results ...
│ │ │ └─Done. (0.109s)
│ │ ├─Calling selftest_path_integrity ...
│ │ │ └─Done. (0.071s)
│ │ ├─Calling selftest_board_power ...
│ │ │ ├─Emulating a normal run ...
│ │ │ ├─POWERON (test-ub-power)
│ │ │ ├─UBOOT (test-ub-power)
│ │ │ ├─POWEROFF (test-ub-power)
│ │ │ ├─Emulating a failing run ...
│ │ │ ├─POWERON (test-ub-power)
│ │ │ ├─UBOOT (test-ub-power)
│ │ │ ├─raise TestException()
│ │ │ ├─POWEROFF (test-ub-power)
│ │ │ └─Done. (0.088s)
│ │ ├─Calling selftest_board_uboot ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─Calling selftest_machine_shell ...
│ │ │ │ ├─Testing command output ...
│ │ │ │ ├─Testing return codes ...
│ │ │ │ ├─Testing env vars ...
│ │ │ │ └─Done. (0.019s)
│ │ │ └─Done. (0.068s)
│ │ ├─Calling selftest_board_uboot_noab ...
│ │ │ ├─UBOOT (test-ub-noab)
│ │ │ ├─Calling selftest_machine_shell ...
│ │ │ │ ├─Testing command output ...
│ │ │ │ ├─Testing return codes ...
│ │ │ │ ├─Testing env vars ...
│ │ │ │ └─Done. (0.015s)
│ │ │ └─Done. (0.055s)
│ │ ├─Calling selftest_board_linux ...
│ │ │ └─Skipped: No board available
│ │ ├─Calling selftest_board_linux_uboot ...
│ │ │ ├─Testing without UB ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─LINUX (test-board-linux-ub)
│ │ │ ├─Testing with UB ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─LINUX (test-board-linux-ub)
│ │ │ └─Done. (0.117s)
│ │ ├─Calling selftest_board_linux_standalone ...
│ │ │ ├─Testing without UB ...
│ │ │ ├─LINUX (test-board-linux-standalone)
│ │ │ ├─Testing with UB ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ └─Done. (0.088s)
│ │ ├─Calling selftest_board_linux_nopw ...
│ │ │ ├─Testing without UB ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─LINUX (test-board-linux-ub_nopw)
│ │ │ ├─Testing with UB ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─LINUX (test-board-linux-ub_nopw)
│ │ │ └─Done. (0.127s)
│ │ ├─Calling selftest_board_linux_bad_console ...
│ │ │ └─Skipped: board-linux bad console test is not implemented
│ │ ├─Calling selftest_with_lab ...
│ │ │ ├─Calling selftest_decorated_lab ...
│ │ │ │ └─Done. (0.057s)
│ │ │ ├─Calling selftest_decorated_lab ...
│ │ │ │ └─Done. (0.006s)
│ │ │ └─Done. (0.069s)
│ │ ├─Calling selftest_with_uboot ...
│ │ │ ├─Calling selftest_decorated_uboot ...
│ │ │ │ ├─UBOOT (test-ub)
│ │ │ │ └─Done. (0.089s)
│ │ │ ├─Calling selftest_decorated_uboot ...
│ │ │ │ ├─UBOOT (test-ub)
│ │ │ │ └─Done. (0.048s)
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─Calling selftest_decorated_uboot ...
│ │ │ │ └─Done. (0.004s)
│ │ │ └─Done. (0.183s)
│ │ ├─Calling selftest_with_linux ...
│ │ │ ├─Calling selftest_decorated_linux ...
│ │ │ │ ├─UBOOT (test-ub)
│ │ │ │ ├─LINUX (test-board-linux-ub)
│ │ │ │ └─Done. (0.110s)
│ │ │ ├─Calling selftest_decorated_linux ...
│ │ │ │ ├─UBOOT (test-ub)
│ │ │ │ ├─LINUX (test-board-linux-ub)
│ │ │ │ └─Done. (0.066s)
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─LINUX (test-board-linux-ub)
│ │ │ ├─Calling selftest_decorated_linux ...
│ │ │ │ └─Done. (0.004s)
│ │ │ └─Done. (0.236s)
│ │ ├─────────────────────────────────────────
│ │ │ Success: 21/21 tests passed
│ │ └─Done. (1.404s)
│ ├─Warning: Some child processes are still running:
│ │ UID PID COMMAND
│ │ 1500 15022 sleep 10
│ │
│ ├─Warning: Some child processes are still running:
│ │ UID PID COMMAND
│ │ 1500 15022 sleep 10
│ │
│ ├─Warning: Some child processes are still running:
│ │ UID PID COMMAND
│ │ 1500 15022 sleep 10
│ │
│ ├─Warning: Some child processes are still running:
│ │ UID PID COMMAND
│ │ 1500 15022 sleep 10
│ │
│ ├─Warning: Some child processes are still running:
│ │ UID PID COMMAND
│ │ 1500 15022 sleep 10
│ │
│ ├─Warning: Some child processes are still running:
│ │ UID PID COMMAND
│ │ 1500 15022 sleep 10
│ │
│ ├─Warning: Some child processes are still running:
│ │ UID PID COMMAND
│ │ 1500 15022 sleep 10
│ │
│ └─Fail. (3.257s)
├─Exception:
│ Traceback (most recent call last):
│ File "/home/sglass/.local/lib/python3.7/site-packages/tbot-0.8.0-py3.7.egg/tbot/main.py", line 318, in main
│ func(**params)
│ File "/home/sglass/.local/lib/python3.7/site-packages/tbot-0.8.0-py3.7.egg/tbot/decorators.py", line 65, in wrapped
│ result = tc(*args, **kwargs)
│ File "/home/sglass/.local/lib/python3.7/site-packages/tbot-0.8.0-py3.7.egg/tbot/tc/selftest/init.py", line 89, in selftest
│ lab=lh,
│ File "/home/sglass/.local/lib/python3.7/site-packages/tbot-0.8.0-py3.7.egg/tbot/machine/machine.py", line 137, in exit
│ self._cx.exit(*args)
│ File "/usr/lib/python3.7/contextlib.py", line 524, in exit
│ raise exc_details[1]
│ File "/usr/lib/python3.7/contextlib.py", line 509, in exit
│ if cb(*exc_details):
│ File "/usr/lib/python3.7/contextlib.py", line 377, in _exit_wrapper
│ return cm_exit(cm, exc_type, exc, tb)
│ File "/home/sglass/.local/lib/python3.7/site-packages/tbot-0.8.0-py3.7.egg/tbot/machine/channel/channel.py", line 478, in exit
│ self.close()
│ File "/home/sglass/.local/lib/python3.7/site-packages/tbot-0.8.0-py3.7.egg/tbot/machine/channel/channel.py", line 456, in close
│ self._c.close()
│ File "/home/sglass/.local/lib/python3.7/site-packages/tbot-0.8.0-py3.7.egg/tbot/machine/channel/subprocess.py", line 106, in close
│ raise Exception("not done")
│ Exception: not done
├─────────────────────────────────────────
└─FAILURE (3.685s)

[Rahix]

Also, is there a quick way to run the local tbot rather than installing it each time?

Yes! After uninstalling your current version with

$ pip3 uninstall --user tbot

you can reinstall it in develop mode:

$ python3 setup.py develop --user

This way, the installation is just a symlink back to the tree you checked out so any changes you do will immediately apply.

---

Thanks for the log, I will take a closer within the next days ... 'Normal' use of tbot should not be affected by this weirdness, from what I can tell.

[Rahix]

I am not able to reproduce your issue, but from the log I have had a guess what might be going on ... I wrote a patch which should help (And apparently GitHub does not allow attaching patches ._.). Can you try applying it?

From f32394fee700b3b863b0bee55f35ddb9b9897871 Mon Sep 17 00:00:00 2001
From: Rahix <[email protected]>
Date: Mon, 2 Dec 2019 11:52:00 +0100
Subject: [PATCH] selftest: Fix a `sleep` still sleeping after test end

Kill a `sleep` subprocess in the selftests to ensure it does not live
past the test end where it would need to be killed by the session
ending.  Ideally, the subprocess channel would be able to properly kill
all child processes but this far I was not able to come up with a 100%
watertight solution.  By preventing this sleep from sleeping too long,
we will at least not have the selftests failing because of it.

Reported-by: Simon Glass <[email protected]>
Signed-off-by: Harald Seiler <[email protected]>
---
 tbot/tc/selftest/machine.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/tbot/tc/selftest/machine.py b/tbot/tc/selftest/machine.py
index 5d5519c84dcf..88351bc5ec5d 100644
--- a/tbot/tc/selftest/machine.py
+++ b/tbot/tc/selftest/machine.py
@@ -173,6 +173,13 @@ def selftest_machine_shell(m: typing.Union[linux.LinuxShell, board.UBootShell])
                 f"Command took {t2 - t1}s (max 9s). Sleep was not sent to background"
             )
 
+            # Kill the sleep process.  In some circumstances, tbot does not
+            # seem to be able to kill all child processes of a subprocess
+            # channel.  To prevent this from leading to issues, kill the sleep
+            # right here instead of relying on tbot to do so correctly at the
+            # very end.      Tracking-Issue: Rahix/tbot#13
+            m.exec("kill", linux.Raw("%%"), linux.Then, "wait", linux.Raw("%%"))
+
         if "control" in cap:
             out = m.exec0(
                 "false", linux.AndThen, "echo", "FOO", linux.OrElse, "echo", "BAR"
-- 
2.20.1

[sjg20]

Hi,

Sorry for the delay. Yes that fixes it. Without the patch it failed 7 out of 10 times. With it 0 out of 10.

Regards,
Simon

[Rahix]

Fixed in f32394f ("selftest: Fix a sleep still sleeping after test end").