GithubHelp home page GithubHelp logo

rahix / tbot Goto Github PK

View Code? Open in Web Editor NEW
82.0 10.0 20.0 5.9 MB

Automation/Testing tool for Embedded Linux Development

Home Page: https://tbot.tools

License: GNU General Public License v3.0

Shell 1.02% Python 98.13% HTML 0.85%
embedded-linux automation developer-tools

tbot's Introduction

tbot
Python 3.8 Checked with mypy Code style: black tbot selftest CI
Embedded Test & Automation Tool

Welcome to tbot! tbot automates the development workflow for embedded systems software. This automation can then also be used for running tests against real hardware, even in CI.

At its core, tbot is a library for interacting with remote hosts over various connections. For example, a target board can be accessed via serial console. Or a TFTP-server via SSH. tbot allows managing all these connections in "parallel". This means, you can orchestrate complex sequences of interaction between them.

At the moment, the main focus of tbot lies in embedded Linux systems. Support for other systems is definitely intended to be added, too.

Most info about tbot can be found in its documentation at https://tbot.tools. You can also join our mailing list, tbot AT lists.denx.de.


Installation

pip3 install --user -U git+https://github.com/rahix/[email protected]

If you haven't done it already, you need to add ~/.local/bin to your $PATH.

Completions

tbot supports command line completions. Install them with the following commands:

curl --create-dirs -L -o ~/.local/lib/tbot/completions.sh https://github.com/Rahix/tbot/raw/master/completions.sh
echo "source ~/.local/lib/tbot/completions.sh" >>~/.bashrc

Usecase Examples

To show what tbot can help you with, here are a few simple example usecases:

Boot into Linux and run a few commands over serial console

@tbot.testcase
def test_linux_simple():
    # request serial connection to Linux on the board
    with tbot.ctx.request(tbot.role.BoardLinux) as lnx:
        # now we can run commands
        lnx.exec0("uname", "-a")

        # or, for example, read a file from the target
        cmdline = (lnx.fsroot / "proc" / "cmdline").read_text()

Define custom bootloader commands to boot Linux

class CustomBoardLinux(board.LinuxUbootConnector, board.LinuxBootLogin, linux.Bash):
    username = "root"
    password = None

    def do_boot(self, uboot):
        # set `autoload` env-var to false to prevent automatic DHCP-boot
        uboot.env("autoload", "false")

        # get an IP-address
        uboot.exec0("dhcp")

        # download kernel + initramfs from TFTP server
        loadaddr = 0x82000000
        uboot.exec0("tftp", hex(loadaddr), f"{tftp_ip}:{kernel_image_path}")

        # and boot it!
        return uboot.boot("bootm", hex(loadaddr))

Network speed test between a board and server

@tbot.testcase
def test_ethernet_speed():
    with tbot.ctx() as cx:
        # boot into Linux on the board and acquire a shell-session
        lnx = cx.request(tbot.role.BoardLinux)

        # use ssh to connect to a network server to test against
        lh = cx.request(tbot.role.LabHost)

        # start iperf server
        with lh.run("iperf", "-s") as iperf_server:
            # and display its output while waiting for startup
            iperf_server.read_until_timeout(2)

            # now run iperf client on DUT
            tx_report = lnx.exec0("iperf", "-c", server_ip)

            # exit the server with CTRL-C
            tbot.log.message("Server Output:")
            iperf_server.sendcontrol("C")
            iperf_server.terminate0()

Contributing

Help is really appreciated! Please take a look at tbot's contribution guidelines for more info. If you are unsure about anything, please open an issue or consult the mailing list first!

License

tbot is licensed under the GNU General Public License v3.0 or later. See LICENSE for more info.

tbot's People

Contributors

bauer-alex-174 avatar cmhe avatar demonpo avatar hsdenx avatar jneuhauser avatar locomotiveviaduct avatar mebel-christ avatar mike8 avatar offdroid avatar rahix avatar saimen avatar sjg20 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tbot's Issues

Failure on sefttest

This happens sometimes, about one time in two on a two-core Intel(R) Celeron(R) CPU N3060 @ 1.60GHz. Oddly enough I cannot make it happen on a 16-core AMD.

9632c7a (HEAD -> master, origin/master, origin/machine-v2, origin/HEAD) Fix deprecated types in annotations

sglass@kea:/vid/software/tbot$ tbot selftest
tbot starting ...
├─Calling selftest ...
│ ├─Calling testsuite ...
│ │ ├─Calling selftest_failing ...
│ │ │ ├─Calling inner ...
│ │ │ │ └─Fail. (0.000s)
│ │ │ └─Done. (0.001s)
│ │ ├─Calling selftest_skipping ...
│ │ │ ├─Calling inner ...
│ │ │ │ └─Skipped: This test is skipped on purpose
│ │ │ └─Done. (0.001s)
│ │ ├─Calling selftest_uname ...
│ │ │ └─Done. (0.004s)
│ │ ├─Calling selftest_user ...
│ │ │ └─Done. (0.002s)
│ │ ├─Calling selftest_machine_reentrant ...
│ │ │ └─Done. (0.002s)
│ │ ├─Calling selftest_machine_labhost_shell ...
│ │ │ ├─Calling selftest_machine_shell ...
│ │ │ │ ├─Testing command output ...
│ │ │ │ ├─Testing return codes ...
│ │ │ │ ├─Testing env vars ...
│ │ │ │ ├─Testing redirection (and weird paths) ...
│ │ │ │ ├─Testing subshell ...
│ │ │ │ └─Done. (0.062s)
│ │ │ ├─Calling selftest_machine_channel ...
│ │ │ │ └─Skipped: Channel tests need to be reimplemented for machine-v2
│ │ │ ├─Calling selftest_machine_channel ...
│ │ │ │ └─Skipped: Channel tests need to be reimplemented for machine-v2
│ │ │ └─Done. (0.172s)
│ │ ├─Calling selftest_machine_ssh_shell ...
│ │ │ ├─Calling check_minisshd ...
│ │ │ │ └─Done. (0.004s)
│ │ │ └─Skipped: dropbear is not installed so ssh can't be tested
│ │ ├─Calling selftest_machine_sshlab_shell ...
│ │ │ ├─Calling check_minisshd ...
│ │ │ │ └─Done. (0.004s)
│ │ │ └─Skipped: dropbear is not installed so ssh can't be tested
│ │ ├─Calling selftest_path_stat ...
│ │ │ ├─Setting up test files ...
│ │ │ ├─Checking existence ...
│ │ │ ├─Checking file modes ...
│ │ │ ├─Checking file modes on nonexistent files ...
│ │ │ ├─Checking stat results ...
│ │ │ └─Done. (0.075s)
│ │ ├─Calling selftest_path_integrity ...
│ │ │ └─Done. (0.081s)
│ │ ├─Calling selftest_board_power ...
│ │ │ ├─Emulating a normal run ...
│ │ │ ├─POWERON (test-ub-power)
│ │ │ ├─UBOOT (test-ub-power)
│ │ │ ├─POWEROFF (test-ub-power)
│ │ │ ├─Emulating a failing run ...
│ │ │ ├─POWERON (test-ub-power)
│ │ │ ├─UBOOT (test-ub-power)
│ │ │ ├─raise TestException()
│ │ │ ├─POWEROFF (test-ub-power)
│ │ │ └─Done. (0.118s)
│ │ ├─Calling selftest_board_uboot ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─Calling selftest_machine_shell ...
│ │ │ │ ├─Testing command output ...
│ │ │ │ ├─Testing return codes ...
│ │ │ │ ├─Testing env vars ...
│ │ │ │ └─Done. (0.014s)
│ │ │ └─Done. (0.076s)
│ │ ├─Calling selftest_board_uboot_noab ...
│ │ │ ├─UBOOT (test-ub-noab)
│ │ │ ├─Calling selftest_machine_shell ...
│ │ │ │ ├─Testing command output ...
│ │ │ │ ├─Testing return codes ...
│ │ │ │ ├─Testing env vars ...
│ │ │ │ └─Done. (0.014s)
│ │ │ └─Done. (0.063s)
│ │ ├─Calling selftest_board_linux ...
│ │ │ └─Skipped: No board available
│ │ ├─Calling selftest_board_linux_uboot ...
│ │ │ ├─Testing without UB ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─LINUX (test-board-linux-ub)
│ │ │ ├─Testing with UB ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─LINUX (test-board-linux-ub)
│ │ │ └─Done. (0.115s)
│ │ ├─Calling selftest_board_linux_standalone ...
│ │ │ ├─Testing without UB ...
│ │ │ ├─LINUX (test-board-linux-standalone)
│ │ │ ├─Testing with UB ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ └─Done. (0.112s)
│ │ ├─Calling selftest_board_linux_nopw ...
│ │ │ ├─Testing without UB ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─LINUX (test-board-linux-ub_nopw)
│ │ │ ├─Testing with UB ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─LINUX (test-board-linux-ub_nopw)
│ │ │ └─Done. (0.132s)
│ │ ├─Calling selftest_board_linux_bad_console ...
│ │ │ └─Skipped: board-linux bad console test is not implemented
│ │ ├─Calling selftest_with_lab ...
│ │ │ ├─Calling selftest_decorated_lab ...
│ │ │ │ └─Done. (0.057s)
│ │ │ ├─Calling selftest_decorated_lab ...
│ │ │ │ └─Done. (0.005s)
│ │ │ └─Done. (0.066s)
│ │ ├─Calling selftest_with_uboot ...
│ │ │ ├─Calling selftest_decorated_uboot ...
│ │ │ │ ├─UBOOT (test-ub)
│ │ │ │ └─Done. (0.108s)
│ │ │ ├─Calling selftest_decorated_uboot ...
│ │ │ │ ├─UBOOT (test-ub)
│ │ │ │ └─Done. (0.065s)
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─Calling selftest_decorated_uboot ...
│ │ │ │ └─Done. (0.004s)
│ │ │ └─Done. (0.225s)
│ │ ├─Calling selftest_with_linux ...
│ │ │ ├─Calling selftest_decorated_linux ...
│ │ │ │ ├─UBOOT (test-ub)
│ │ │ │ ├─LINUX (test-board-linux-ub)
│ │ │ │ └─Done. (0.117s)
│ │ │ ├─Calling selftest_decorated_linux ...
│ │ │ │ ├─UBOOT (test-ub)
│ │ │ │ ├─LINUX (test-board-linux-ub)
│ │ │ │ └─Done. (0.070s)
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─LINUX (test-board-linux-ub)
│ │ │ ├─Calling selftest_decorated_linux ...
│ │ │ │ └─Done. (0.004s)
│ │ │ └─Done. (0.251s)
│ │ ├─────────────────────────────────────────
│ │ │ Success: 21/21 tests passed
│ │ └─Done. (1.524s)
│ └─Done. (1.589s)
├─────────────────────────────────────────
└─SUCCESS (2.016s)
sglass@kea:/vid/software/tbot$ tbot selftest
tbot starting ...
├─Calling selftest ...
│ ├─Calling testsuite ...
│ │ ├─Calling selftest_failing ...
│ │ │ ├─Calling inner ...
│ │ │ │ └─Fail. (0.000s)
│ │ │ └─Done. (0.001s)
│ │ ├─Calling selftest_skipping ...
│ │ │ ├─Calling inner ...
│ │ │ │ └─Skipped: This test is skipped on purpose
│ │ │ └─Done. (0.001s)
│ │ ├─Calling selftest_uname ...
│ │ │ └─Done. (0.004s)
│ │ ├─Calling selftest_user ...
│ │ │ └─Done. (0.002s)
│ │ ├─Calling selftest_machine_reentrant ...
│ │ │ └─Done. (0.001s)
│ │ ├─Calling selftest_machine_labhost_shell ...
│ │ │ ├─Calling selftest_machine_shell ...
│ │ │ │ ├─Testing command output ...
│ │ │ │ ├─Testing return codes ...
│ │ │ │ ├─Testing env vars ...
│ │ │ │ ├─Testing redirection (and weird paths) ...
│ │ │ │ ├─Testing subshell ...
│ │ │ │ └─Done. (0.051s)
│ │ │ ├─Calling selftest_machine_channel ...
│ │ │ │ └─Skipped: Channel tests need to be reimplemented for machine-v2
│ │ │ ├─Calling selftest_machine_channel ...
│ │ │ │ └─Skipped: Channel tests need to be reimplemented for machine-v2
│ │ │ └─Done. (0.159s)
│ │ ├─Calling selftest_machine_ssh_shell ...
│ │ │ ├─Calling check_minisshd ...
│ │ │ │ └─Done. (0.005s)
│ │ │ └─Skipped: dropbear is not installed so ssh can't be tested
│ │ ├─Calling selftest_machine_sshlab_shell ...
│ │ │ ├─Calling check_minisshd ...
│ │ │ │ └─Done. (0.004s)
│ │ │ └─Skipped: dropbear is not installed so ssh can't be tested
│ │ ├─Calling selftest_path_stat ...
│ │ │ ├─Setting up test files ...
│ │ │ ├─Checking existence ...
│ │ │ ├─Checking file modes ...
│ │ │ ├─Checking file modes on nonexistent files ...
│ │ │ ├─Checking stat results ...
│ │ │ └─Done. (0.080s)
│ │ ├─Calling selftest_path_integrity ...
│ │ │ └─Done. (0.099s)
│ │ ├─Calling selftest_board_power ...
│ │ │ ├─Emulating a normal run ...
│ │ │ ├─POWERON (test-ub-power)
│ │ │ ├─UBOOT (test-ub-power)
│ │ │ ├─POWEROFF (test-ub-power)
│ │ │ ├─Emulating a failing run ...
│ │ │ ├─POWERON (test-ub-power)
│ │ │ ├─UBOOT (test-ub-power)
│ │ │ ├─raise TestException()
│ │ │ ├─POWEROFF (test-ub-power)
│ │ │ └─Done. (0.107s)
│ │ ├─Calling selftest_board_uboot ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─Calling selftest_machine_shell ...
│ │ │ │ ├─Testing command output ...
│ │ │ │ ├─Testing return codes ...
│ │ │ │ ├─Testing env vars ...
│ │ │ │ └─Done. (0.014s)
│ │ │ └─Done. (0.063s)
│ │ ├─Calling selftest_board_uboot_noab ...
│ │ │ ├─UBOOT (test-ub-noab)
│ │ │ ├─Calling selftest_machine_shell ...
│ │ │ │ ├─Testing command output ...
│ │ │ │ ├─Testing return codes ...
│ │ │ │ ├─Testing env vars ...
│ │ │ │ └─Done. (0.015s)
│ │ │ └─Done. (0.053s)
│ │ ├─Calling selftest_board_linux ...
│ │ │ └─Skipped: No board available
│ │ ├─Calling selftest_board_linux_uboot ...
│ │ │ ├─Testing without UB ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─LINUX (test-board-linux-ub)
│ │ │ ├─Testing with UB ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─LINUX (test-board-linux-ub)
│ │ │ └─Done. (0.152s)
│ │ ├─Calling selftest_board_linux_standalone ...
│ │ │ ├─Testing without UB ...
│ │ │ ├─LINUX (test-board-linux-standalone)
│ │ │ ├─Testing with UB ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ └─Done. (0.102s)
│ │ ├─Calling selftest_board_linux_nopw ...
│ │ │ ├─Testing without UB ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─LINUX (test-board-linux-ub_nopw)
│ │ │ ├─Testing with UB ...
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─LINUX (test-board-linux-ub_nopw)
│ │ │ └─Done. (0.125s)
│ │ ├─Calling selftest_board_linux_bad_console ...
│ │ │ └─Skipped: board-linux bad console test is not implemented
│ │ ├─Calling selftest_with_lab ...
│ │ │ ├─Calling selftest_decorated_lab ...
│ │ │ │ └─Done. (0.063s)
│ │ │ ├─Calling selftest_decorated_lab ...
│ │ │ │ └─Done. (0.005s)
│ │ │ └─Done. (0.071s)
│ │ ├─Calling selftest_with_uboot ...
│ │ │ ├─Calling selftest_decorated_uboot ...
│ │ │ │ ├─UBOOT (test-ub)
│ │ │ │ └─Done. (0.133s)
│ │ │ ├─Calling selftest_decorated_uboot ...
│ │ │ │ ├─UBOOT (test-ub)
│ │ │ │ └─Done. (0.053s)
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─Calling selftest_decorated_uboot ...
│ │ │ │ └─Done. (0.004s)
│ │ │ └─Done. (0.233s)
│ │ ├─Calling selftest_with_linux ...
│ │ │ ├─Calling selftest_decorated_linux ...
│ │ │ │ ├─UBOOT (test-ub)
│ │ │ │ ├─LINUX (test-board-linux-ub)
│ │ │ │ └─Done. (0.116s)
│ │ │ ├─Calling selftest_decorated_linux ...
│ │ │ │ ├─UBOOT (test-ub)
│ │ │ │ ├─LINUX (test-board-linux-ub)
│ │ │ │ └─Done. (0.082s)
│ │ │ ├─UBOOT (test-ub)
│ │ │ ├─LINUX (test-board-linux-ub)
│ │ │ ├─Calling selftest_decorated_linux ...
│ │ │ │ └─Done. (0.004s)
│ │ │ └─Done. (0.261s)
│ │ ├─────────────────────────────────────────
│ │ │ Success: 21/21 tests passed
│ │ └─Done. (1.545s)
│ └─Fail. (3.137s)
├─Exception:
│ Traceback (most recent call last):
│ File "/home/sglass/.local/lib/python3.7/site-packages/tbot-0.8.0-py3.7.egg/tbot/main.py", line 318, in main
│ func(**params)
│ File "/home/sglass/.local/lib/python3.7/site-packages/tbot-0.8.0-py3.7.egg/tbot/decorators.py", line 65, in wrapped
│ result = tc(*args, **kwargs)
│ File "/home/sglass/.local/lib/python3.7/site-packages/tbot-0.8.0-py3.7.egg/tbot/tc/selftest/init.py", line 89, in selftest
│ lab=lh,
│ File "/home/sglass/.local/lib/python3.7/site-packages/tbot-0.8.0-py3.7.egg/tbot/machine/machine.py", line 137, in exit
│ self._cx.exit(*args)
│ File "/usr/lib/python3.7/contextlib.py", line 524, in exit
│ raise exc_details[1]
│ File "/usr/lib/python3.7/contextlib.py", line 509, in exit
│ if cb(*exc_details):
│ File "/usr/lib/python3.7/contextlib.py", line 377, in _exit_wrapper
│ return cm_exit(cm, exc_type, exc, tb)
│ File "/home/sglass/.local/lib/python3.7/site-packages/tbot-0.8.0-py3.7.egg/tbot/machine/channel/channel.py", line 478, in exit
│ self.close()
│ File "/home/sglass/.local/lib/python3.7/site-packages/tbot-0.8.0-py3.7.egg/tbot/machine/channel/channel.py", line 456, in close
│ self._c.close()
│ File "/home/sglass/.local/lib/python3.7/site-packages/tbot-0.8.0-py3.7.egg/tbot/machine/channel/subprocess.py", line 99, in close
│ raise Exception("not done")
│ Exception: not done
├─────────────────────────────────────────
└─FAILURE (3.569s)

Early load config when integrating tbot into pytest

I am currently working on a test setup with tbot + pytest, but I am facing some challenges with the proposed skeleton integration. As you know the execution of pytest can be roughly boiled down to the following two stages:

  1. Collection stage: pytest searches all the tests, parameters, fixtures,... that make up the test configurations
  2. Execution stage: pytest actually executes the collected test configurations

By design pytest does not execute fixtures during collection stage which in turn makes it impossible to interact with tbot during that stage since the tbot integration is solely done as a fixture. I want to propose an alternative integration mechanism to make it possible to interact with tbot during collection stage

Use case

First up I want to outline my use case why I want to interact with tbot during collection stage in the first place with the following (minimal) working example. I have several configuration files that may look like this:

import tbot
from tbot.machine import board, connector

class DummyBoard(connector.SubprocessConnector, board.Board):

    @property
    def interfaces(self):
        return ["interface0", "interface1"]

def register_machines(ctx):
    ctx.register(DummyBoard, tbot.role.Board)

I want to manage variable interface types and amounts at the board level. For that regard I use a property in the Board class that returns how many instances of a certain type this Board has. For simplicity reasons in this example the interface is modeled by a simple string.

Assume there is a test that checks whether a single interface (of a certain type) works as expected. Naturally I want to run this test for every single interface instance present on a Board. With pytest this is possible by using the pytest.mark.parametrize decorator like so:

import pytest
import tbot

def _interfaces():
    with tbot.ctx.request(tbot.role.Board) as b:
        return b.interfaces

@pytest.mark.parametrize("interface", _interfaces())
def test_interface(interface):
    print(interface)

With the currently described integration mechanism this results in tbot.error.MachineNotFoundError: no machine found for <class 'tbot.role.Board'> since parametrize runs during the collection stage and hence when requesting the Board context in _interfaces() the fixture that loads the config was not executed yet.

Proposal

To make this work I propose the following change to conftest.py:

import pytest
import tbot
from tbot import newbot

def pytest_addoption(parser, pluginmanager):
    parser.addoption("--tbot-config", action="append", default=[], dest="tbot_configs")
    parser.addoption("--tbot-flag", action="append", default=[], dest="tbot_flags")

def pytest_sessionstart(session):
    # Only register configuration when nobody else did so already.
    if not tbot.ctx.is_active():
        # Register flags
        for flag in session.config.option.tbot_flags:
            tbot.flags.add(flag)
        # Register configuration
        for config in session.config.option.tbot_configs:
            newbot.load_config(config, tbot.ctx)

@pytest.fixture(scope="session", autouse=True)
def tbot_ctx():
    with tbot.ctx:
        # Configure the context for keep_alive (so machines can be reused
        # between tests).  reset_on_error_by_default will make sure test
        # failures lead to a powercycle of the DUT anyway.
        with tbot.ctx.reconfigure(keep_alive=True, reset_on_error_by_default=True):
            # Tweak the standard output logging options
            with tbot.log.with_verbosity(tbot.log.Verbosity.STDOUT, nesting=1):
                yield tbot.ctx

Move the "configuration registration" step to the pytest_sessionstart hook. This loads the tbot configuration before the collection stage runs and makes it possible to parametrize a test in the described fashion:

$ pytest -v --tbot-config config.dummy
=================================================================== test session starts ====================================================================
...
collected 2 items                                                                                                                                          

tests/test_interface.py::test_interface[interface0] PASSED                                                                                           [ 50%]
tests/test_interface.py::test_interface[interface1] PASSED                                                                                           [100%]

==================================================================== 2 passed in 0.02s =====================================================================

Are there some Gotchas I am missing here? Otherwise I would happily supply a PR to the doc. Feedback on that approach is greatly appreciated.

Fix kernel-log clobbering serial console

If kernel-logging is set to be verbose, kernel log-messages will clobber the serial console and lead to incorrect test results or prevent proper board initialization.

tbot should at least be able to acquire the shell and maybe provide a config option that allows turning off kernel logging via sysctl.

Base exception classed raised

It looks like there are 40 cases where raise Exception is used rather than a subclass of Exception. This makes it impossible to catch Tbot-specific errors. Ideally, every exception raised should inherit from tbot.error.TbotException

Tbot Selftest fails in virtual environment

The tbot selftest selftest_machine_labhost_shell fails if tbot is executed in a virtual environment. Running tbot outside the virtual environment is successful.

Steps to reproduce:

  1. Have tbot installed with pip3 install -U --user git+https://github.com/rahix/[email protected]
  2. Create a virtual environment python -m venv venv
  3. Activate the virtual environment source venv/bin/activate
  4. Run tbot selftest

Tested with Python version Python 3.8.5 and Python 3.9.5

Sample output:

$ tbot selftest
tbot starting ...
├─Calling selftest ...
│   ├─Calling testsuite ...
│   │   ├─Calling selftest_failing ...
│   │   │   ├─Calling inner ...
│   │   │   │   └─Fail. (0.000s)
│   │   │   └─Done. (0.000s)
│   │   ├─Calling selftest_skipping ...
│   │   │   ├─Calling inner ...
│   │   │   │   └─Skipped: This test is skipped on purpose
│   │   │   └─Done. (0.000s)
│   │   ├─Calling selftest_uname ...
│   │   │   └─Done. (0.002s)
│   │   ├─Calling selftest_user ...
│   │   │   └─Done. (0.001s)
│   │   ├─Calling selftest_machine_reentrant ...
│   │   │   └─Done. (0.000s)
│   │   ├─Calling selftest_machine_labhost_shell ...
│   │   │   ├─Calling selftest_machine_shell ...
│   │   │   │   ├─Testing command output ...
│   │   │   │   ├─Testing return codes ...
│   │   │   │   ├─Testing env vars ...
│   │   │   │   ├─Testing redirection (and weird paths) ...
│   │   │   │   ├─99895
│   │   │   │   ├─Testing subshell ...
│   │   │   │   ├─Testing mach.run() ...
│   │   │   │   └─Fail. (0.034s)
│   │   │   └─Fail. (0.034s)
│   │   ├─Calling selftest_machine_ssh_shell ...
│   │   │   ├─Calling check_minisshd ...
│   │   │   │   └─Done. (0.002s)
│   │   │   └─Skipped: dropbear is not installed so ssh can't be tested
│   │   ├─Calling selftest_machine_sshlab_shell ...
│   │   │   ├─Calling check_minisshd ...
│   │   │   │   └─Done. (0.002s)
│   │   │   └─Skipped: dropbear is not installed so ssh can't be tested
│   │   ├─Calling selftest_path_stat ...
│   │   │   ├─Setting up test files ...
│   │   │   ├─Checking existence ...
│   │   │   ├─Checking file modes ...
│   │   │   ├─Checking file modes on nonexistent files ...
│   │   │   ├─Checking stat results ...
│   │   │   └─Done. (0.022s)
│   │   ├─Calling selftest_path_integrity ...
│   │   │   └─Done. (0.047s)
│   │   ├─Calling selftest_path_files ...
│   │   │   ├─Testing text file access ...
│   │   │   ├─Testing binary file access ...
│   │   │   ├─Test reading/writing invalid file ...
│   │   │   └─Done. (0.024s)
│   │   ├─Calling selftest_board_power ...
│   │   │   ├─Emulating a normal run ...
│   │   │   ├─POWERON (test-ub-power)
│   │   │   ├─UBOOT (test-ub-power)
│   │   │   ├─POWEROFF (test-ub-power)
│   │   │   ├─Emulating a failing run ...
│   │   │   ├─POWERON (test-ub-power)
│   │   │   ├─UBOOT (test-ub-power)
│   │   │   ├─raise TestException()
│   │   │   ├─POWEROFF (test-ub-power)
│   │   │   └─Done. (0.246s)
│   │   ├─Calling selftest_board_uboot ...
│   │   │   ├─UBOOT (test-ub)
│   │   │   ├─Calling selftest_machine_shell ...
│   │   │   │   ├─Testing command output ...
│   │   │   │   ├─Testing return codes ...
│   │   │   │   ├─Testing env vars ...
│   │   │   │   └─Done. (0.008s)
│   │   │   └─Done. (0.237s)
│   │   ├─Calling selftest_board_uboot_noab ...
│   │   │   ├─UBOOT (test-ub-noab)
│   │   │   ├─Calling selftest_machine_shell ...
│   │   │   │   ├─Testing command output ...
│   │   │   │   ├─Testing return codes ...
│   │   │   │   ├─Testing env vars ...
│   │   │   │   └─Done. (0.009s)
│   │   │   └─Done. (0.431s)
│   │   ├─Calling selftest_board_linux ...
│   │   │   └─Skipped: No board available
│   │   ├─Calling selftest_board_linux_uboot ...
│   │   │   ├─Testing without UB ...
│   │   │   ├─UBOOT (test-ub)
│   │   │   ├─LINUX (test-board-linux-ub)
│   │   │   ├─Testing with UB ...
│   │   │   ├─UBOOT (test-ub)
│   │   │   ├─LINUX (test-board-linux-ub)
│   │   │   └─Done. (0.866s)
│   │   ├─Calling selftest_board_linux_standalone ...
│   │   │   ├─Testing without UB ...
│   │   │   ├─LINUX (test-board-linux-standalone)
│   │   │   ├─Testing with UB ...
│   │   │   ├─UBOOT (test-ub)
│   │   │   └─Done. (1.446s)
│   │   ├─Calling selftest_board_linux_nopw ...
│   │   │   ├─Testing without UB ...
│   │   │   ├─UBOOT (test-ub)
│   │   │   ├─LINUX (test-board-linux-ub_nopw)
│   │   │   ├─Testing with UB ...
│   │   │   ├─UBOOT (test-ub)
│   │   │   ├─LINUX (test-board-linux-ub_nopw)
│   │   │   └─Done. (0.463s)
│   │   ├─Calling selftest_board_linux_bad_console ...
│   │   │   └─Skipped: board-linux bad console test is not implemented
│   │   ├─Calling selftest_with_lab ...
│   │   │   ├─Calling selftest_decorated_lab ...
│   │   │   │   └─Done. (0.022s)
│   │   │   ├─Calling selftest_decorated_lab ...
│   │   │   │   └─Done. (0.001s)
│   │   │   └─Done. (0.024s)
│   │   ├─Calling selftest_with_uboot ...
│   │   │   ├─Calling selftest_decorated_uboot ...
│   │   │   │   ├─UBOOT (test-ub)
│   │   │   │   └─Done. (0.440s)
│   │   │   ├─Calling selftest_decorated_uboot ...
│   │   │   │   ├─UBOOT (test-ub)
│   │   │   │   └─Done. (0.620s)
│   │   │   ├─UBOOT (test-ub)
│   │   │   ├─Calling selftest_decorated_uboot ...
│   │   │   │   └─Done. (0.001s)
│   │   │   └─Done. (1.480s)
│   │   ├─Calling selftest_with_linux ...
│   │   │   ├─Calling selftest_decorated_linux ...
│   │   │   │   ├─UBOOT (test-ub)
│   │   │   │   ├─LINUX (test-board-linux-ub)
│   │   │   │   └─Done. (0.051s)
│   │   │   ├─Calling selftest_decorated_linux ...
│   │   │   │   ├─UBOOT (test-ub)
│   │   │   │   ├─LINUX (test-board-linux-ub)
│   │   │   │   └─Done. (1.033s)
│   │   │   ├─UBOOT (test-ub)
│   │   │   ├─LINUX (test-board-linux-ub)
│   │   │   ├─Calling selftest_decorated_linux ...
│   │   │   │   └─Done. (0.002s)
│   │   │   └─Done. (1.114s)
│   │   ├─────────────────────────────────────────
│   │   │ Failure: 1/22 tests failed
│   │   │ 
│   │   ├─selftest_machine_labhost_shell:
│   │   │ Traceback (most recent call last):
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/tc/__init__.py", line 43, in testsuite
│   │   │     test(**kwargs)
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/decorators.py", line 62, in wrapped
│   │   │     return tc(*args, **kwargs)
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/tc/selftest/machine.py", line 54, in selftest_machine_labhost_shell
│   │   │     selftest_machine_shell(lh)
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/decorators.py", line 62, in wrapped
│   │   │     return tc(*args, **kwargs)
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/tc/selftest/machine.py", line 258, in selftest_machine_shell
│   │   │     bs.terminate0()
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/machine/linux/util.py", line 160, in terminate0
│   │   │     retcode, output = self.terminate()
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/machine/linux/util.py", line 176, in terminate
│   │   │     next(self._cmd_context)
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/machine/linux/bash.py", line 206, in cmd_context
│   │   │     retcode = int(proxy_ch.read_until_prompt())
│   │   │ ValueError: invalid literal for int() with base 10: '2004l\rexit\necho $?\n'
│   │   │ 
│   │   └─Fail. (6.444s)
│   └─Fail. (6.665s)
├─Exception:
│   Traceback (most recent call last):
│     File "/home/nou/.local/lib/python3.9/site-packages/tbot/main.py", line 345, in main
│       func(**params)
│     File "/home/nou/.local/lib/python3.9/site-packages/tbot/decorators.py", line 62, in wrapped
│       return tc(*args, **kwargs)
│     File "/home/nou/.local/lib/python3.9/site-packages/tbot/tc/selftest/__init__.py", line 80, in selftest
│       tc.testsuite(
│     File "/home/nou/.local/lib/python3.9/site-packages/tbot/decorators.py", line 62, in wrapped
│       return tc(*args, **kwargs)
│     File "/home/nou/.local/lib/python3.9/site-packages/tbot/tc/__init__.py", line 62, in testsuite
│       raise Exception(f"{len(errors)}/{len(args)} tests failed")
│   Exception: 1/22 tests failed
├─────────────────────────────────────────
└─FAILURE (6.721s)

How to access role.Board from role.BoardUboot and role.BoardLinux?

Hello,

I have a configuration like below where I want to access properties of my board configuration from my Uboot or Linux configuration. See the method init() in the class MyUboot. The configuration below works because board.Connector provides _board but by convention we should not rely on properties with underscore prefix.

import tbot
from tbot.machine import board, channel, connector

class MyBoard(
    connector.ConsoleConnector,
    board.PowerControl,
    board.Board,
):
    device_id = 0
    device_ip = "10.20.30.43"
    netmask = "255.255.255.0"
    gateway_ip = "10.20.30.254"
    relay_ip = "10.20.30.41"
	
    def connect(self, mach) -> channel.Channel:
        return mach.open_channel("picocom", "-b", "115200", "/dev/ttyUSB0")
	
    def poweron(self) -> None:
        self.host.exec0("curl", "http://{self.relay_ip}/io.cgi?relay={device_id}?state=1")
	
    def poweroff(self) -> None:
        self.host.exec0("curl", "http://{self.relay_ip}/io.cgi?relay={device_id}?state=0")


class MyUboot(
    board.Connector,
    board.UBootAutobootIntercept,
    board.UBootShell,
):
    def init(self) -> None:
        self.env("ipaddr", self._board.device_ip)
        self.env("netmask", self._board.netmask)
        self.env("gatewayip", self._board.gateway_ip)


def register_machines(ctx: tbot.Context) -> None:
    ctx.register(MyBoard, tbot.role.Board)
    ctx.register(MyUboot, tbot.role.BoardUBoot)

Is there an intended way of accessing these objects like the context API for tests or do we need to introduce something new?

Thanks in advance, Johann

test/py test-framework hangs on some tests



~/Documents/dmo-system-test$ python3 -m pytest -s --tbot-config config.serial_local_config  test_cases/test_uboot.py --html=report.html
========================================================= test session starts ==========================================================
platform linux -- Python 3.8.10, pytest-7.3.1, pluggy-1.0.0
rootdir: /home/adnanelhammoudi/Documents/dmo-system-test
plugins: metadata-3.0.0, xdist-3.5.0, html-4.1.1
collected 1 item                                                                                                                       

test_cases/test_uboot.py │   ├─Calling test_run_testpy ...
│   │   ├─[local] picocom --quiet -b 115200 -r -l /dev/ttyUSB2
│   │   ├─POWERON (imx8)
│   │   ├─UBOOT (device-u-boot)
│   │   │    <> 
│   │   │    <> 
│   │   │    <> <INTERRUPT>
│   │   │    <> => reboot
│   │   │    <> Unknown command 'reboot' - try 'help'
│   │   │    <> => cat
│   │   │    <> Unknown command 'cat' - try 'help'
│   │   │    <> => reset
│   │   │    <> resetting ...
│   │   │    <> 
│   │   │    <> 
│   │   │    <> U-Boot 2020.04.YodaPro-sma-20221215 (Dec 15 2022 - 20:08:00 +0000)
│   │   │    <> 
│   │   │    <> CPU:   NXP i.MX8DXL RevA1 A35 at 1200 MHz at 45C
│   │   │    <> 
│   │   │    <> DRAM:  1022 MiB
│   │   │    <> MMC:   FSL_SDHC: 0
│   │   │    <> Loading Environment from MMC... OK
│   │   │    <> In:    serial
│   │   │    <> Out:   serial
│   │   │    <> Err:   serial
│   │   │    <> Model: SMA YODAPRO
│   │   │    <> Board: VS-YODA-PRO-COM
│   │   │    <> DISP:  not found
│   │   │    <> sc_seco_v2x_build_info: res:8
│   │   │    <> 
│   │   │    <>  BuildInfo: 
│   │   │    <>   - SCFW cae076f8, SECO-FW b6fcc2c6, IMX-MKIMAGE f3335e39, ATF lf-5.15
│   │   │    <>   - U-Boot 2020.04.YodaPro-sma-20221215 
│   │   │    <> 
│   │   │    <> flash target is MMC:0
│   │   │    <> Net:   eth0: ethernet@5b050000
│   │   │    <> Fastboot: Normal
│   │   │    <> Normal Boot
│   │   │    <> Press SPACE to abort autoboot in 3 seconds
│   │   │    <> =>  
│   │   ├─[local] mkdir -p /home/adnanelhammoudi/Documents
│   │   ├─Calling uboot_testpy ...
│   │   │   ├─[local] bash --norc --noprofile
│   │   │   ├─Calling uboot_setup_testhooks ...
│   │   │   │   ├─[local] test -d /home/adnanelhammoudi/Documents/uboot-testpy-tbot
│   │   │   │   ├─Creating FIFOs ...
│   │   │   │   ├─[local] rm -rf /home/adnanelhammoudil/Documents/uboot-testpy-tbot/fifo_console_send
│   │   │   │   ├─[local] mkfifo /home/adnanelhammoudi/Documents/uboot-testpy-tbot/fifo_console_send
│   │   │   │   ├─[local] rm -rf /home/adnanelhammoudi/Documents/uboot-testpy-tbot/fifo_console_recv
│   │   │   │   ├─[local] mkfifo /home/adnanelhammoudi/Documents/uboot-testpy-tbot/fifo_console_recv
│   │   │   │   ├─[local] rm -rf /home/adnanelhammoudi/Documents/uboot-testpy-tbot/fifo_commands
│   │   │   │   ├─[local] mkfifo /home/adnanelhammoudi/Documents/uboot-testpy-tbot/fifo_commands
│   │   │   │   ├─[local] cat /home/adnanelhammoudi/Documents/uboot-testpy-tbot/tbot-scripts.sha256
│   │   │   │   │    ## 2d30892b61eb713ce9413e06c4f2a0cd00d2a74b6b8c2ac6624e1e49909b1897
│   │   │   │   ├─Hooks are up to date, skipping deployment ...
│   │   │   │   ├─Adding hooks to $PATH ...
│   │   │   │   ├─[local] echo " ${PATH}"
│   │   │   │   │    ##  /home/adnanelhammoudi/Documents/dmo-system-test/scripts/env/bin:/home/adnanelhammoudi/.vscode-server/bin/0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2/bin/remote-cli:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
│   │   │   │   ├─[local] export PATH=/home/adnanelhammoudil/Documents/uboot-testpy-tbot:/home/adnanelhammoudi/Documents/dmo-system-test/scripts/env/bin:/home/adnanelhammoudi/.vscode-server/bin/0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2/bin/remote-cli:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
│   │   │   │   ├─Open console & command channels ...
│   │   │   │   ├─[local] /home/adnanelhammoudi/Documents/uboot-testpy-tbot/tbot-console
│   │   │   │   ├─[local] /home/adnanelhammoudi/Documents/uboot-testpy-tbot/tbot-commands
│   │   │   │   └─Done. (0.030s)
│   │   │   ├─[local] test -e /home/adnanelhammoudi/Documents/u-boot/.config
│   │   │   ├─[local] test -e /home/adnanelhammoudi/Documents/u-boot/include/autoconf.mk
│   │   │   ├─[local] printf %s '' >/home/adnanelhammoudi/Documents/u-boot/test/py/u_boot_boardenv_tbot_imx8.py
│   │   │   ├─[local] cd /home/adnanelhammoudi/Documents/u-boot
│   │   │   ├─[local] ./test/py/test.py --build-dir . --board-type tbot-imx8
│   │   │   │    ## +u-boot-test-flash tbot-imx8 na
│   │   │   │    ## ================================= test session starts ==================================
│   │   │   │    ## platform linux -- Python 3.8.10, pytest-7.3.1, pluggy-1.0.0
│   │   │   │    ## rootdir: /home/adnanelhammoudi/Documents/u-boot/test/py
│   │   │   │    ## configfile: pytest.ini
│   │   │   │    ## plugins: metadata-3.0.0, xdist-3.5.0, html-4.1.1
│   │   │   │    ## collected 429 items                                                                    
│   │   │   │    ## 
│   │   │   │    ## test/py/tests/test_000_version.py 
│   │   │   └─Fail. (31.088s)
│   │   ├─POWEROFF (imx8q)
│   │   └─Fail. (34.379s)
FAILED

============================================================================= FAILURES =============================================================================
_________________________________________________________________________ test_run_testpy __________________________________________________________________________

    @tbot.testcase
    def test_run_testpy():
        with tbot.ctx() as cx:
            lb = cx.request(tbot.role.LocalHost)
            b = cx.request(tbot.role.Board)
            ub = cx.request(tbot.role.BoardUBoot)
            build_dir = lb.workdir
>           uboot.testpy(
            build_dir,
            boardenv=BOARDENV,
            board=b,
            uboot=ub,
            testpy_args=["-k", "not (clean  or dm or efi or env or hashes or gpt or help or hush or gpio or net or md or part or pinmux or shell or sleep)"],)

test_cases/test_uboot.py:14: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../.local/lib/python3.8/site-packages/tbot/decorators.py:92: in wrapped
    return tc(*args, **kwargs)
../../.local/lib/python3.8/site-packages/tbot_contrib/uboot/_testpy.py:343: in testpy
    b.poweron()
config/serial_local_config.py:46: in poweron
    self.ch.sendcontrol("C")
../../.local/lib/python3.8/site-packages/tbot/machine/channel/channel.py:705: in sendcontrol
    self.write(bytes([num]), _ignore_blacklist=True)
../../.local/lib/python3.8/site-packages/tbot/machine/channel/channel.py:342: in write
    bytes_written = self._c.write(buf[cursor:])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tbot.machine.channel.channel.ChannelBorrowed object at 0x7fe7dbdbeb50>, buf = b'\x03'

    def write(self, buf: bytes) -> int:
>       raise self.exception()
E       tbot.error.ChannelBorrowedError: channel is currently borrowed by another machine

../../.local/lib/python3.8/site-packages/tbot/machine/channel/channel.py:159: ChannelBorrowedError
------------------------- Generated html report: file:///home/adnanelhammoudi%40digitalgateamg.local/Documents/dmo-system-test/report.html -------------------------
===================================================================== short test summary info ======================================================================
FAILED test_cases/test_uboot.py::test_run_testpy - tbot.error.ChannelBorrowedError: channel is currently borrowed by another machine
======================================================================== 1 failed in 34.48s =======================================================================

At least Python 3.8 is required by tbot/newbot

Running tbot or newbot always aborts with following exception:

$ newbot
Traceback (most recent call last):
  File "/home/mike/.local/bin/newbot", line 8, in <module>
    sys.exit(main())
  File "/home/mike/.local/lib/python3.6/site-packages/tbot/newbot.py", line 186, in main
    parser = build_parser()
  File "/home/mike/.local/lib/python3.6/site-packages/tbot/newbot.py", line 148, in build_parser
    parser.add_argument("-c", "--config", nargs=1, action="extend", default=[])
  File "/usr/lib/python3.6/argparse.py", line 1346, in add_argument
    raise ValueError('unknown action "%s"' % (action_class,))
ValueError: unknown action "extend"

The reason for this is that action="extend" is used from argparse.py. This action has been introduced in Python 3.8 according to the documentation. tbot documentation say the minimum required Python version is 3.6, however.

This should be fixed either by still supporting Python 3.6 and therefore change the implementation, or, at least and much easier, by adopting the docu.

Paramiko SSHException: not found in known_hosts

I'm having some trouble connecting to a Raspberry Pi as a Lab config. I'm getting the follow error when I try to run the interactive_lab command. I can connect using the command ssh 172.16.100.119 without a password just fine. This connection is also in the known_hosts file.

paramiko.ssh_exception.SSHException: Server '172.16.100.119' not found in known_hosts

ssh_config

Host 172.16.100.119
  HostName 172.16.100.119
  User pi
  IdentityFile /home/drewwestrick/.ssh/rasp_pi

tc.py

import tbot
from tbot.machine import connector, linux

class AwesomeLab(
    connector.ParamikoConnector,
    linux.Bash,
    linux.Lab,
):
    name = "172.16.100.119"
    hostname = "172.16.100.119"
    username = "pi"

    @property
    def workdir(self):
        return linux.Workdir.athome(self, "tbot-workdir")


# tbot will check for `LAB`, don't forget to set it!
LAB = AwesomeLab
```c

PyPI Package

Is it possible to that you publish this project also to PyPI?

Allow access to boot log

Currently there is no way to access the boot log from within a testcase. This should be possible to allow testing certain functionality.

Support workdir relative to tbot CWD

The working directory of the lab may be defined as a relative path:

def workdir(self):
    linux.Workdir.static(self, f"tbot-workdir")

This leads to a failure in kconfig_set_enabled:

│   ├─Calling kconfig_set_enabled ...
│   │   ├─Enabling CONFIG_CMD_BOOTEFI_SELFTEST option ...
│   │   ├─[rpi2] sed -i '/^\(# \)\?CONFIG_CMD_BOOTEFI_SELFTEST\(=[ym]\| is not set\)$/cCONFIG_CMD_BOOTEFI_SELFTEST=y' tbot-workdir/uboot-orangepipc/.config
│   │   │    ## sed: can't read tbot-workdir/uboot-orangepipc/.config: No such file or directory
│   │   └─Fail. (0.014s)
│   └─Fail. (36.598s)

sed is executed in the work directory. So the work directory path must not be prepended.

Compatibility of `machine.linux.Path` with unaware code

I have a need [1] to convert a tbot Path object to a PurePosixPath or to a string to pass onto other code. Currently I use super(machine.linux.Path, obj).__str__() which is "less than ideal" IMHO.

Is there another way? What does not work:

  • p.parent / p.name: The parent is also a tbot Path (makes sense) so that result is the same.
  • p.as_posix(): Has the name attribute of the host prepended to the string.

Shouldn't PurePath.as_posix() be implemented in machine.linux.Path to provide the expected behavior? Currently it is only
implemented in PurePath and calls __str__(), which is overridden by machine.linux.Path and which prepends the name of the host the path is associated with. Implementing machine.linux.Path.__str__() that way makes perfect sense, but it breaks as_posix().

Maybe as_posix() shall raise an Exception if the associated host is not isinstance(self.host, connector.SubprocessConnector) but otherwise return the expected string?

[1] My device under test (DUT) runs a webserver. The tests shall talk to it using Python requests over HTTPS. To avoid HTTPS certificate warnings, a known certificate and private key is deployed onto the DUT using SSH. It is convenient to use a tbot Path object for this in the context of (1) checking the certificate validity period using localhost.exec0(openssl...), (2) creating the certificate using localhost.exec0(openssl...), and deploying the certificate using tbot's shell.copy(). But finally the path must be handed over to the requests module, and it cannot be a tbot Path object for obvious reasons.

tbot installation

I am a beginner in linux. I was trying to use tbot for testing. After installation as mentioned in documentation, tbot selftest returns tbot: command not found.
completions.sh file is added to bashrc file.

Can you please help ?
Thanks !

Timeout when waiting for a temination of a subprocess is hardcoded

Hello,

We are using a VM as a subprocess for testing our embedded system. When the tests complete and Tbot is trying to shut down the subprocess channel, it sends a TERM signal to the VM process and waits ~ 1.27 seconds for the VM process to exit.

In some occasions, it takes more than 1.27 seconds for the VM process to exit which triggers tbot.error.TbotException("some subprocess(es) did not stop") exception.

It would be great to be able to configure or override the timeout waiting for a subprocess to exit in def close(self) in tbot/machine/channel/subprocess.py.

thanks,
Yaniv

Processes not being terminated after SSH connection

Even after #88 there are still some file-descriptors that appear to remain open (even though they should be closed) when I connect to a DUT with the SSHConnector. I use this setup:

config/eval.py:

from tbot import role
from tbot.machine import board, connector, linux
from tbot.machine.linux import auth

class EvalBoard_Linux(connector.SSHConnector, linux.Bash):

    @property
    def authenticator(self) -> auth.Authenticator:
        return auth.PasswordAuthenticator("<password>")
    
    @property
    def hostname(self) -> str:
        return "<ip-address>"
    
    @property
    def username(self) -> str:
        return "<username>"


def register_machines(ctx):
    ctx.register(EvalBoard_Linux, role.BoardLinux)

tc/test_fd.py:

import os
import tbot
import time

from tbot import role, log

@tbot.testcase
def test_fds():
    log.message(f"tbot PID: {os.getpid()}")
    time.sleep(5)

    for i in range(5):
        log.message(f"RUN {i}")
        with tbot.ctx.request(role.BoardLinux) as lnx:
            lnx.exec0("echo", "Hello Linux!")
        log.message(f'Open FDs: {sorted(set(os.listdir("/proc/self/fd")))}')

I execute the setup without any additional configuration like this: newbot -c config.eval tc.test_fd.test_fds
This is the output:

tbot starting ...
├─Calling test_fds ...
│   ├─tbot PID: 6006
│   ├─RUN 0
│   ├─[local] sshpass -p <password> ssh -p 22 <username>@<ip-address>
│   ├─[eval-board_-linux] echo 'Hello Linux!'
│   │    ## Hello Linux!
│   ├─Open FDs: ['0', '1', '103', '11', '2', '26', '27', '28', '29', '3', '30', '39', '4', '6', '7']
│   ├─RUN 1
│   ├─[local] sshpass -p <password> ssh -p 22 <username>@<ip-address>
│   ├─[eval-board_-linux] echo 'Hello Linux!'
│   │    ## Hello Linux!
│   ├─Open FDs: ['0', '1', '103', '11', '2', '26', '27', '28', '29', '3', '30', '39', '4', '6', '7', '8', '9']
│   ├─RUN 2
│   ├─[local] sshpass -p <password> ssh -p 22 <username>@<ip-address>
│   ├─[eval-board_-linux] echo 'Hello Linux!'
│   │    ## Hello Linux!
│   ├─Open FDs: ['0', '1', '10', '103', '11', '12', '2', '26', '27', '28', '29', '3', '30', '39', '4', '6', '7', '8', '9']
│   ├─RUN 3
│   ├─[local] sshpass -p <password> ssh -p 22 <username>@<ip-address>
│   ├─[eval-board_-linux] echo 'Hello Linux!'
│   │    ## Hello Linux!
│   ├─Open FDs: ['0', '1', '10', '103', '11', '12', '13', '14', '2', '26', '27', '28', '29', '3', '30', '39', '4', '6', '7', '8', '9']
│   ├─RUN 4
│   ├─[local] sshpass -p <password> ssh -p 22 <username>@<ip-address>
│   ├─[eval-board_-linux] echo 'Hello Linux!'
│   │    ## Hello Linux!
│   ├─Open FDs: ['0', '1', '10', '103', '11', '12', '13', '14', '15', '16', '2', '26', '27', '28', '29', '3', '30', '39', '4', '6', '7', '8', '9']
│   └─Done. (8.901s)
├─────────────────────────────────────────
└─SUCCESS (8.934s)

This time it seems that some processes remain active, after the connection is closed. htop shows this:
tbot

@Rahix can you confirm this happens as well on your end?

Proposed solution

Change the yield step in the current _connect() method of SSHConnector like this:

_ch = h.ch.take()
yield _ch
_ch.close()

This properly closes all remaining channels / processes and results in the following test run:

tbot starting ...
├─Calling test_fds ...
│   ├─tbot PID: 9308
│   ├─RUN 0
│   ├─[local] sshpass -p <password> ssh -p 22 <username>@<ip-address>
│   ├─[eval-board_-linux] echo 'Hello Linux!'
│   │    ## Hello Linux!
│   ├─Open FDs: ['0', '1', '103', '11', '2', '26', '27', '28', '29', '3', '31', '33', '35', '39', '4']
│   ├─RUN 1
│   ├─[local] sshpass -p <password> ssh -p 22 <username>@<ip-address>
│   ├─[eval-board_-linux] echo 'Hello Linux!'
│   │    ## Hello Linux!
│   ├─Open FDs: ['0', '1', '103', '11', '2', '26', '27', '28', '29', '3', '31', '33', '35', '39', '4']
│   ├─RUN 2
│   ├─[local] sshpass -p <password> ssh -p 22 <username>@<ip-address>
│   ├─[eval-board_-linux] echo 'Hello Linux!'
│   │    ## Hello Linux!
│   ├─Open FDs: ['0', '1', '103', '11', '2', '26', '27', '28', '29', '3', '31', '33', '35', '39', '4']
│   ├─RUN 3
│   ├─[local] sshpass -p <password> ssh -p 22 <username>@<ip-address>
│   │    ## Hello Linux!
│   ├─Open FDs: ['0', '1', '103', '11', '2', '26', '27', '28', '29', '3', '31', '33', '35', '39', '4']
│   ├─RUN 4
│   ├─[local] sshpass -p <password> ssh -p 22 <username>@<ip-address>
│   ├─[eval-board_-linux] echo 'Hello Linux!'
│   │    ## Hello Linux!
│   ├─Open FDs: ['0', '1', '103', '11', '2', '26', '27', '28', '29', '3', '31', '33', '35', '39', '4']
│   └─Done. (8.902s)
├─────────────────────────────────────────
└─SUCCESS (8.937s)

Add docker machine for building

A lot of projects are build inside docker containers nowadays. TBot should have a builtin docker machine for this purpose.

Detail virtualenv installation

Global installation as it is currently described is convenient but bears problems when updating tbot: Newer tbot might not be 100% compatible with old projects.

Instead, one could install tbot into a project-specific virtualenv and pin the version to something that's known to work. This approach should be detailed in the documentation as well.

newbot CLI cannot generate log files

The tbot CLI supports a --log option, which generates a log file for offline analysis. This is useful for comparing results between runs, or when retrieving logs from a pipeline run.

newbot does not support this option. Given the tbot and newbot CLIs appear to use the same underlying logging infrastructure, it seems this could be implemented by copying a small amount of CLI logic from tbot to newbot.

This was tested with #78, which generated the same log files, module timestamp differences.

When running tests via SSH connector and SSH connection breaks due to device crash TBot hangs waiting indefinitely for SSH

Hello,

Sometimes, when running tests on our device the device crashes and the SSH connection breaks. When this happens TBot waits Indefinitely.
We are using the connector.SSHConnector connector, and have added the following options to ssh via ssh_config property: ['ConnectTimeout=5', 'ServerAliveInterval=1', 'ServerAliveCountMax=5']. With these options, SSH detects that the device has failed, but TBot does not detect it and continues to wait for an input from SSH.

Please see below an example of output. After "exit" is printed, TBot is waiting indefinitely:

│   ├─[local] ssh -o BatchMode=yes -o StrictHostKeyChecking=no -p 22 -o ConnectTimeout=5 -o ServerAliveInterval=1 -o ServerAliveCountMax=5 [email protected]
│   ├─[nr-sim-linux] uname -a
│   │    ## 14:54:13.717558    11 dwmac.cc:178] read to A-N registers not supported (0xc4))(will warn only once)
│   │    ## uname -a
│   │    ## Linux abc 5.18.0-g38e0cf23fe26 #1 SMP PREEMPT Thu Aug 11 09:07:41 UTC 2022 aarch64 GNU/Linux
│   ├─[nr-sim-linux-ssh] ls -l
│   │    ## Timeout, server 192.168.5.12 not responding.
│   │    ## exit

Is there a way to detect that the SSH connection lost and fail the test in such case?

thanks,
Yaniv

OpenSBI

For the RISC-V Maixduino board I will need to first build U-Boot and then OpenSBI passing U-Boot as payload:

prepare:
	test -d opensbi || git clone -v \
	https://github.com/riscv/opensbi.git

sbi:
	cd opensbi && make \
	PLATFORM=kendryte/k210 \
	FW_PAYLOAD=y \
	FW_PAYLOAD_OFFSET=0x20000 \
	FW_PAYLOAD_PATH=../u-boot/u-boot-dtb.bin
	kflash/kflash.py -p $(TTY) -b 1500000 -B maixduino \
	opensbi/build/platform/kendryte/k210/firmware/fw_payload.bin
	picocom -b 115200 --send-cmd "sz -vv" $(TTY)

Other RISC-V boards will require to first build OpenSBI and then pass its location to the U-Boot build process (cf. doc/board/emulation/qemu-riscv.rst).

Similar requirements exist for Trusted Firmware on boards like Pine64 A64 LTS where you have to provide TF-A as environment variable BL31.

Should an OpenSBI build testcase be added to tbot or do you consider such work out of scope for your project?

Output form uboot_testpy for test requiring power cycling

The print out of the uboot_testpy target is messed up when sub-tests are requiring power cycling:

│   │    ## test/py/tests/test_efi_loader.py .sss.s│   ├─[rpi2] relay-card off
│   ├─[rpi2] relay-card off
│   ├─[rpi2] sd-mux-ctrl -v 0 -td
│   ├─[rpi2] relay-card on
                                [  0%]
│   │    ## test/py/tests/test_efi_selftest.py .│   ├─[rpi2] relay-card off
│   ├─[rpi2] relay-card off
│   ├─[rpi2] sd-mux-ctrl -v 0 -td
│   ├─[rpi2] relay-card on
.│   ├─[rpi2] relay-card off
│   ├─[rpi2] relay-card off
│   ├─[rpi2] sd-mux-ctrl -v 0 -td
│   ├─[rpi2] relay-card on
.│   ├─[rpi2] relay-card off
│   ├─[rpi2] relay-card off
│   ├─[rpi2] sd-mux-ctrl -v 0 -td
│   ├─[rpi2] relay-card on
.│   ├─[rpi2] relay-card off
│   ├─[rpi2] relay-card off
│   ├─[rpi2] sd-mux-ctrl -v 0 -td
│   ├─[rpi2] relay-card on
.                               [  0%]
│   │    ## test/py/tests/test_env.py ............sss

Can the output of poweron and poweroff be muted here?

[Help] Run a pipe shell command

Hi everyone,

Could you tell me the way to run a pipe shell command with tbot Linux Bash ? For example: dmesg | grep "ABC"

Thanks in advance !

Upgrading from 0.7.x to 0.8.x yields broken installation

On my system (Arch Linux), tbot selftest (or any other tbot invocation) fails with the following error:

[laptop tbot]$ ~/.local/bin/tbot selftest
Traceback (most recent call last):
  File "/home/laptop/.local/bin/tbot", line 8, in <module>
    from tbot.main import main
  File "/home/laptop/.local/lib/python3.8/site-packages/tbot-0.8.3-py3.8.egg/tbot/__init__.py", line 22, in <module>
    from . import selectable
  File "/home/laptop/.local/lib/python3.8/site-packages/tbot-0.8.3-py3.8.egg/tbot/selectable.py", line 18, in <module>
    from tbot.machine import connector, linux, board
  File "/home/laptop/.local/lib/python3.8/site-packages/tbot-0.8.3-py3.8.egg/tbot/machine/__init__.py", line 1, in <module>
    from . import board
  File "/home/laptop/.local/lib/python3.8/site-packages/tbot-0.8.3-py3.8.egg/tbot/machine/board/__init__.py", line 4, in <module>
    from .uboot import UBootShell, UBootAutobootIntercept
  File "/home/laptop/.local/lib/python3.8/site-packages/tbot-0.8.3-py3.8.egg/tbot/machine/board/uboot.py", line 24, in <module>
    from ..linux import special
  File "/home/laptop/.local/lib/python3.8/site-packages/tbot-0.8.3-py3.8.egg/tbot/machine/linux/__init__.py", line 34, in <module>
    from . import build
  File "/home/laptop/.local/lib/python3.8/site-packages/tbot-0.8.3-py3.8.egg/tbot/machine/linux/build/__init__.py", line 1, in <module>
    from .machine import BuildMachine
  File "/home/laptop/.local/lib/python3.8/site-packages/tbot-0.8.3-py3.8.egg/tbot/machine/linux/build/machine.py", line 20, in <module>
    from . import toolchain
  File "/home/laptop/.local/lib/python3.8/site-packages/tbot-0.8.3-py3.8.egg/tbot/machine/linux/build/toolchain.py", line 22, in <module>
    class Toolchain(abc.ABC):
  File "/home/laptop/.local/lib/python3.8/site-packages/tbot-0.8.3-py3.8.egg/tbot/machine/linux/build/toolchain.py", line 26, in Toolchain
    def enable(self, host: linux.LinuxMachine) -> None:
AttributeError: partially initialized module 'tbot.machine.linux' has no attribute 'LinuxMachine' (most likely due to a circular import)

The issue appeared between 0.7.1 (which works) and 0.8.0 (which doesn't work). Latest 0.8.3 doesn't work either.

[laptop tbot]$ python --version
Python 3.8.5

I installed using: python3 setup.py install --user

Am I missing something :)?

Possible bug when auto login is used on the device

Hello.

I have a problem with a case when a board uses an autologin. read_until_prompt method doesn't work if the device keeps sending logs.
For example, if we set login_promt like this ~ # and the input bytearray will become like this ~ # [ 10.364872] startup_sh[0]: IP is: 192.168.80.254 the method will just ignore the string.

My suggestion is to replace endwith with find here

Please, guide me if I'm wrong.

shutil error while copying( fifo_console_send, fifo_console_recv, fifo_commands) in py/tests/test_cleanup_build.py

 ## test/py/tests/test_bind.py ss                                                                     [  0%]
│   │   │   │    ## test/py/tests/test_bootmenu.py s                                                                  [  1%]
│   │   │   │    ## test/py/tests/test_button.py ss                                                                   [  2%]
│   │   │   │    ## test/py/tests/test_cleanup_build.py EE                                                            [  3%]
│   │   │   │    ## test/py/tests/test_event_dump.py s                                                                [  3%]
│   │   │   │    ## test/py/tests/test_extension.py s                                                                 [  4%]
│   │   │   │    ## test/py/tests/test_fit.py s                                                                       [  4%]
│   │   │   │    ## test/py/tests/test_fit_auto_signed.py s                                                           [  5%]
│   │   │   │    ## test/py/tests/test_fit_ecdsa.py s                                                                 [  5%]
│   │   │   │    ## test/py/tests/test_fpga.py ssssssssssssssss                                                       [ 13%]
│   │   │   │    ## test/py/tests/test_handoff.py s                                                                   [ 14%]
│   │   │   │    ## test/py/tests/test_kconfig.py ss                                                                  [ 15%]
│   │   │   │    ## test/py/tests/test_log.py ss                                                                      [ 16%]
│   │   │   │    ## test/py/tests/test_lsblk.py s                                                                     [ 16%]
│   │   │   │    ## test/py/tests/test_of_migrate.py sss                                                              [ 17%]
│   │   │   │    ## test/py/tests/test_ofplatdata.py s                                                                [ 18%]
│   │   │   │    ## test/py/tests/test_pstore.py sss                                                                  [ 19%]
│   │   │   │    ## test/py/tests/test_qfw.py ss                                                                      [ 20%]
│   │   │   │    ## test/py/tests/test_sandbox_exit.py ssss                                                           [ 22%]
│   │   │   │    ## test/py/tests/test_scp03.py s                                                                     [ 23%]
│   │   │   │    ## test/py/tests/test_source.py s                                                                    [ 23%]
│   │   │   │    ## test/py/tests/test_spl.py ss                                                                      [ 24%]
│   │   │   │    ## test/py/tests/test_stackprotector.py s                                                            [ 25%]
│   │   │   │    ## test/py/tests/test_tpm2.py sssssssssss                                                            [ 30%]
│   │   │   │    ## test/py/tests/test_trace.py s                                                                     [ 31%]
│   │   │   │    ## test/py/tests/test_ut.py s                                                                        [ 31%]
│   │   │   │    ## test/py/tests/test_vbe.py s                                                                       [ 32%]
│   │   │   │    ## test/py/tests/test_vbe_vpl.py s                                                                   [ 32%]
│   │   │   │    ## test/py/tests/test_vboot.py ssssssssssssssssssssssssssssss                                        [ 47%]
│   │   │   │    ## test/py/tests/test_vpl.py s                                                                       [ 47%]
│   │   │   │    ## test/py/tests/test_android/test_ab.py s                                                           [ 48%]
│   │   │   │    ## test/py/tests/test_android/test_abootimg.py ss                                                    [ 49%]
│   │   │   │    ## test/py/tests/test_android/test_avb.py ssssss                                                     [ 51%]
│   │   │   │    ## test/py/tests/test_cat/test_cat.py s                                                              [ 52%]
│   │   │   │    ## test/py/tests/test_fs/test_basic.py sssssssssssssssssssssssssssssssssssssss                       [ 71%]
│   │   │   │    ## test/py/tests/test_fs/test_erofs.py s                                                             [ 71%]
│   │   │   │    ## test/py/tests/test_fs/test_ext.py ssssssssssssssssssssssss                                        [ 83%]
│   │   │   │    ## test/py/tests/test_fs/test_mkdir.py ssssssssssss                                                  [ 89%]
│   │   │   │    ## test/py/tests/test_fs/test_symlink.py ssss                                                        [ 91%]
│   │   │   │    ## test/py/tests/test_fs/test_unlink.py ssssssssssssss                                               [ 98%]
│   │   │   │    ## test/py/tests/test_fs/test_squashfs/test_sqfs_load.py s                                           [ 98%]
│   │   │   │    ## test/py/tests/test_fs/test_squashfs/test_sqfs_ls.py s                                             [ 99%]
│   │   │   │    ## test/py/tests/test_semihosting/test_hostfs.py s                                                   [ 99%]
│   │   │   │    ## test/py/tests/test_xxd/test_xxd.py s                                                              [100%]
│   │   │   │    ## 
│   │   │   │    ## ================================================ ERRORS =================================================
│   │   │   │    ## _____________________________________ ERROR at setup of test_clean ______________________________________
│   │   │   │    ## test/py/tests/test_cleanup_build.py:20: in tmp_copy_of_builddir
│   │   │   │    ##     shutil.copytree(
│   │   │   │    ## /usr/lib/python3.8/shutil.py:557: in copytree
│   │   │   │    ##     return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
│   │   │   │    ## /usr/lib/python3.8/shutil.py:513: in _copytree
│   │   │   │    ##     raise Error(errors)
│   │   │   │    ## E   shutil.Error: [('./uboot-testpy-tbot/fifo_commands', '/tmp/pytest-of-adnanelhammoudi/pytest-6/test_clean0/uboot-testpy-tbot/fifo_commands', '`./uboot-testpy-tbot/fifo_commands` is a named pipe'), ('./uboot-testpy-tbot/fifo_console_send', '/tmp/pytest-of-adnanelhammoudi/pytest-6/test_clean0/uboot-testpy-tbot/fifo_console_send', '`./uboot-testpy-tbot/fifo_console_send` is a named pipe'), ('./uboot-testpy-tbot/fifo_console_recv', '/tmp/pytest-of-adnanelhammoudi/pytest-6/test_clean0/uboot-testpy-tbot/fifo_console_recv', '`./uboot-testpy-tbot/fifo_console_recv` is a named pipe')]
│   │   │   │    ## ____________________________________ ERROR at setup of test_mrproper ____________________________________
│   │   │   │    ## test/py/tests/test_cleanup_build.py:20: in tmp_copy_of_builddir
│   │   │   │    ##     shutil.copytree(
│   │   │   │    ## /usr/lib/python3.8/shutil.py:557: in copytree
│   │   │   │    ##     return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
│   │   │   │    ## /usr/lib/python3.8/shutil.py:513: in _copytree
│   │   │   │    ##     raise Error(errors)
│   │   │   │    ## E   shutil.Error: [('./uboot-testpy-tbot/fifo_commands', '/tmp/pytest-of-adnanelhammoudi/pytest-6/test_mrproper0/uboot-testpy-tbot/fifo_commands', '`./uboot-testpy-tbot/fifo_commands` is a named pipe'), ('./uboot-testpy-tbot/fifo_console_send', '/tmp/pytest-of-adnanelhammoudi/pytest-6/test_mrproper0/uboot-testpy-tbot/fifo_console_send', '`./uboot-testpy-tbot/fifo_console_send` is a named pipe'), ('./uboot-testpy-tbot/fifo_console_recv', '/tmp/pytest-of-adnanelhammoudi/pytest-6/test_mrproper0/uboot-testpy-tbot/fifo_console_recv', '`./uboot-testpy-tbot/fifo_console_recv` is a named pipe')]
│   │   │   │    ## ======================================== short test summary info ========================================
│   │   │   │    ## ERROR test/py/tests/test_cleanup_build.py::test_clean - shutil.Error: [('./uboot-testpy-tbot/fifo_comm...
│   │   │   │    ## ERROR test/py/tests/test_cleanup_build.py::test_mrproper - shutil.Error: [('./uboot-testpy-tbot/fifo_c...
│   │   │   │    ## ============================ 204 skipped, 223 deselected, 2 errors in 2.78s =============================
│   │   │   └─Fail. (3.820s)
│   │   ├─POWEROFF (imx8q)
│   │   └─Fail. (5.321s)
FAILED

======================================================================= FAILURES ========================================================================
____________________________________________________________________ test_run_testpy ____________________________________________________________________

    @tbot.testcase
    def test_run_testpy():
        with tbot.ctx() as cx:
            lb = cx.request(tbot.role.LocalHost)
            b = cx.request(tbot.role.Board)
            ub = cx.request(tbot.role.BoardUBoot, exclusive=True)
            build_dir = lb.workdir
>           uboot.testpy(
            build_dir,
            boardenv=BOARDENV,
            board=b,
            uboot=ub,
            testpy_args=["-k", "not (gpio or version or dm or efi or env or hashes or gpt or help or hush or net or md or part or pinmux or shell or sleep)"],)

test_cases/test_uboot.py:14: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.local/lib/python3.8/site-packages/tbot/decorators.py:92: in wrapped
    return tc(*args, **kwargs)
../../.local/lib/python3.8/site-packages/tbot_contrib/uboot/_testpy.py:353: in testpy
    chan_testpy.terminate0()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tbot.machine.linux.util.RunCommandProxy object at 0x7fee57663dd0>

    def terminate0(self) -> str:
        """
        Wait for the command to end **successfully**.
    
        Asserts that the command returned with retcode 0.  If it did not, an
        exception is raised.
    
        :returns: Remaining output of the command until completion.
        :rtype: str
        """
        retcode, output = self.terminate()
        if retcode != 0:
>           raise tbot.error.CommandFailure(
                self._exc_host, self._exc_args, repr=self._cmd
            )
E           tbot.error.CommandFailure: command failed: [Local] ./test/py/test.py --build-dir . --board-type tbot-imx8q -k 'not (gpio or version or dm or efi or env or hashes or gpt or help or hush or net or md or part or pinmux or shell or sleep)'

../../.local/lib/python3.8/site-packages/tbot/machine/linux/util.py:184: CommandFailure

Better error presentation

Currently, when something goes wrong, tbot dumps a stack-trace and a tiny exception message. Even worse, often it is hard to decipher what actually went wrong and how to deal with this.

Instead, tbot should display clean error messages which tell at least the following information:

  • What exactly went wrong (root cause), as good as tbot can know this.
  • Whether this is a bug in user code or an unexpected problem on some remote machine.
  • What are common solutions to this problem/where can a user find more information.

I'm not sure about stack-traces: Making then off-by-default and configurable might be annoying when a problem only occurs rarely and an unlucky user didn't have stack-traces enabled during the run. Always showing the full trace is over-verbose in most situations.

Display SSH connection errors

Right now, when an SSH connection fails, no useful error message is shown. If a user wants to find out why the connection failed, they have to retry in channel-debug mode (newbot -v) to see the error printed by the ssh command.

Ideally, tbot should detect the connection failure and log the error message before raising an exception. This is a bit tricky because the error shows up right in a spot where we also expect data that we don't want to log to console (shell initialization).

A possible solution may be to detect a closed channel during ssh initialization and only log output then. This may have timing problems, though.

Order of requested roles should not matter

Hi,

I am using tbot together with pytest. I recently experienced some strange behavior in tbot when it comes to requesting roles from the tbot.Context.

The tbot.Context is configured in conftest.py with the following settings:

# conftest.py

@pytest.fixture(scope="session", autouse=True)
def tbot_setup():
    tbot.log.LOGFILE = open("log/tbot_test.json", "w")
    tbot.log.VERBOSITY = 3
    tbot.log.NESTING += 1
    yield
    tbot.log.NESTING -= 1


@pytest.fixture(scope="session")
def tbot_context() -> Iterator[tbot.Context]:
    with tbot.Context(keep_alive=True, reset_on_error_by_default=True) as ctx:
        if os.environ.get('CI') == 'true':
            ctx.register(tbot.selectable.LocalLabHost, [tbot.role.LabHost])
        else:
            my_remote_lab.register_machines(ctx)

        my_board.register_machines(ctx)

        yield ctx


@pytest.fixture(scope="module", autouse=True)
def tbot_nesting():
    print("")
    tbot.log.NESTING += 1
    yield
    print("")
    tbot.log.NESTING -= 1

Since I am running the tests in a Gitlab Pipeline, I use the environment variable CI to determine, if the local host should be used as LabHost or if I need to connect to a remote lab. This is done in tbot_context().

When writing tests, I noticed different behavior if I was running the tests in the CI pipeline (i.e. using the local host as LabHost) and on my development machine using the SSHConnector to connect to the same machine used for the tests in the CI pipeline.

So I tracked down the problem with the following two tests where I request either the LabHost or the BoardLinux first:

# test_lh_first.py

import tbot

def test_lh_first(tbot_context: tbot.Context):
    with tbot_context.request(tbot.role.LabHost) as lh:
        lh.exec0("echo", "Hello World")

    with tbot_context.request(tbot.role.BoardLinux) as lnx:
        lnx.exec0("echo", "Hello World")
# test_lnx_first.py

import tbot

def test_lnx_first(tbot_context: tbot.Context):
    with tbot_context.request(tbot.role.BoardLinux) as lnx:
        lnx.exec0("echo", "Hello World")

    with tbot_context.request(tbot.role.LabHost) as lh:
        lh.exec0("echo", "Hello World")

I can run the tests with these commands directly on the lab host:

# pretend that we are in the CI environment to use the local host as LabHost
export CI=true
# run either test_lnx_first.py or test_lh_first.py
python3 -m pytest -v --capture=tee-sys --log-cli-level=INFO -v --color=yes test_lnx_first.py
python3 -m pytest -v --capture=tee-sys --log-cli-level=INFO -v --color=yes test_lh_first.py

The test_lnx_first.py works fine, with the test_lh_first.py I get the following error:

================================================================= short test summary info ==================================================================
ERROR test_lh_first.py::test_lh_first - Exception: trying to de-init a closed instance
=============================================================== 1 passed, 1 error in 39.52s ================================================================

When I run the tests from my development machine and use the SSHConnector to connect to the LabHost I can run the test_lh_first.py test without any errors.

So I wonder if the order in which roles are requested from the context matter when the teardown of the tests is executed and the roles are de-initialized.

My current workaround is to request the BoardLinux role first and the LabHost afterwards, but this means I have to wait until the board is powered up before I can use the LabHost which can be impractical in some cases where configuration needs to be done on the LabHost before the board is requested and powered up.

Am I missing something here so tbot should be used in a different way like I do?

Best regards
NoU

test reports

Hi,

I used to work through Jenkins and see all nice GUI reports and statistics about my running tests.
Is there a way to connect new TBOT to Jenkins?
How do I see reports/statistics of my regressions?
Any snapshot examples somewhere I can view?

Regards,
Noam

Tbot for windows

Currently its not possible to start tbot tests from a windows machine.

After a quick analysation I came up with following remarks:

  • The packages: pty, fcntl and termios are not available on Windows.
  • In order to make workarounds following classes/functions would need to be changed: SubprocessChannelIO and attach_interactive

Do you see any other concerns?

uboot_testpy: POWERON and POWEROFF in the wrong sequence

I have created scripts to manage my test environment. They are available at
https://github.com/xypron/tbotrunner/tree/orangepipc

Target interactive_uboot runs as expected.

Target uboot_testpy powers the board on before collecting the unit tests and powers it off before executing the unit tests. Obviously this leads to failures in all tests.

A file 'test/py/u_boot_boardenv_tbot_OrangePi PC.py' is created. But that file does not match the name of the defconfig. Why should the name property of the Board class be used here?

Files u-boot-test-console u-boot-test-flash u-boot-test-quit u-boot-test-reset are missing.

user@rpi2:~/workspace/tbotrunner$ tbot -l labconfig.py -b boardconfig.py -vv uboot_testpy
tbot starting ...
├─Calling uboot_testpy ...
│   ├─Message: MyLabHost.build
│   ├─[rpi2] bash --norc --noprofile
│   ├─Calling uboot_setup_testhooks ...
│   │   ├─[rpi2] mkdir -p /tmp/tbot-workdir
│   │   ├─[rpi2] test -d /tmp/tbot-workdir/uboot-testpy-tbot
│   │   ├─Creating FIFOs ...
│   │   ├─[rpi2] rm -rf /tmp/tbot-workdir/uboot-testpy-tbot/fifo_console_send
│   │   ├─[rpi2] mkfifo /tmp/tbot-workdir/uboot-testpy-tbot/fifo_console_send
│   │   ├─[rpi2] rm -rf /tmp/tbot-workdir/uboot-testpy-tbot/fifo_console_recv
│   │   ├─[rpi2] mkfifo /tmp/tbot-workdir/uboot-testpy-tbot/fifo_console_recv
│   │   ├─[rpi2] rm -rf /tmp/tbot-workdir/uboot-testpy-tbot/fifo_commands
│   │   ├─[rpi2] mkfifo /tmp/tbot-workdir/uboot-testpy-tbot/fifo_commands
│   │   ├─[rpi2] cat /tmp/tbot-workdir/uboot-testpy-tbot/tbot-scripts.sha256
│   │   │    ## 2d30892b61eb713ce9413e06c4f2a0cd00d2a74b6b8c2ac6624e1e49909b1897
│   │   ├─Hooks are up to date, skipping deployment ...
│   │   ├─Adding hooks to $PATH ...
│   │   ├─[rpi2] echo " ${PATH}"
│   │   │    ##  /home/user/.local/bin:/home/user/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
│   │   ├─[rpi2] export PATH=/tmp/tbot-workdir/uboot-testpy-tbot:/home/user/.local/bin:/home/user/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
│   │   ├─Open console & command channels ...
│   │   ├─[rpi2] /tmp/tbot-workdir/uboot-testpy-tbot/tbot-console
│   │   ├─[rpi2] /tmp/tbot-workdir/uboot-testpy-tbot/tbot-commands
│   │   └─Done. (0.111s)
│   ├─Calling uboot_checkout ...
│   │   ├─Builder: orangepipc
│   │   ├─[rpi2] test -d /tmp/tbot-workdir/uboot-orangepipc/.git
│   │   ├─[rpi2] git -C /tmp/tbot-workdir/uboot-orangepipc fetch
│   │   │    ## remote: Enumerating objects: 1, done.
│   │   │    ## remote: Counting objects: 100% (1/1), done.
│   │   │    ## remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
│   │   │    ## Unpacking objects: 100% (1/1), done.
│   │   │    ## From https://gitlab.denx.de/u-boot/u-boot
│   │   │    ##    0570938e3c..0437cc4155  master     -> origin/master
│   │   └─Done. (1.822s)
│   ├─[rpi2] test -e /tmp/tbot-workdir/uboot-orangepipc/.config
│   ├─[rpi2] test -e /tmp/tbot-workdir/uboot-orangepipc/include/autoconf.mk
│   ├─[rpi2] picocom -b 115200 /dev/serial/by-path/platform-3f980000.usb-usb-0:1.1.3:1.0-port0
│   ├─POWERON (OrangePi PC)
│   ├─[rpi2] relay-card off
│   ├─[rpi2] sd-mux-ctrl -v 0 -td
│   ├─[rpi2] relay-card on
│   ├─UBOOT (my-board-u-boot)
│   │    <> picocom v3.1
│   │    <> 
│   │    <> port is        : /dev/serial/by-path/platform-3f980000.usb-usb-0:1.1.3:1.0-port0
│   │    <> flowcontrol    : none
│   │    <> baudrate is    : 115200
│   │    <> parity is      : none
│   │    <> databits are   : 8
│   │    <> stopbits are   : 1
│   │    <> escape is      : C-a
│   │    <> local echo is  : no
│   │    <> noinit is      : no
│   │    <> noreset is     : no
│   │    <> hangup is      : no
│   │    <> nolock is      : no
│   │    <> send_cmd is    : sz -vv
│   │    <> receive_cmd is : rz -vv -E
│   │    <> imap is        : 
│   │    <> omap is        : 
│   │    <> emap is        : crcrlf,delbs,
│   │    <> logfile is     : none
│   │    <> initstring     : none
│   │    <> exit_after is  : not set
│   │    <> exit is        : no
│   │    <> 
│   │    <> Type [C-a] [C-h] to see available commands
│   │    <> Terminal ready
│   │    <> 
│   │    <> U-Boot SPL 2020.10-00607-g0570938e3c (Oct 09 2020 - 19:50:13 +0000)
│   │    <> DRAM: 1024 MiB
│   │    <> Trying to boot from MMC1
│   │    <> 
│   │    <> 
│   │    <> U-Boot 2020.10-00607-g0570938e3c (Oct 09 2020 - 19:50:13 +0000) Allwinner Technology
│   │    <> 
│   │    <> CPU:   Allwinner H3 (SUN8I 1680)
│   │    <> Model: Xunlong Orange Pi PC
│   │    <> DRAM:  1 GiB
│   │    <> MMC:   mmc@1c0f000: 0
│   │    <> Loading Environment from FAT... *** Warning - bad CRC, using default environment
│   │    <> 
│   │    <> In:    serial
│   │    <> Out:   serial
│   │    <> Err:   serial
│   │    <> Net:   phy interface0
│   │    <> eth0: ethernet@1c30000
│   │    <> starting USB...
│   │    <> Bus usb@1c1a000: USB EHCI 1.00
│   │    <> Bus usb@1c1a400: USB OHCI 1.0
│   │    <> Bus usb@1c1b000: USB EHCI 1.00
│   │    <> Bus usb@1c1b400: USB OHCI 1.0
│   │    <> Bus usb@1c1c000: USB EHCI 1.00
│   │    <> Bus usb@1c1c400: USB OHCI 1.0
│   │    <> Bus usb@1c1d000: USB EHCI 1.00
│   │    <> Bus usb@1c1d400: USB OHCI 1.0
│   │    <> scanning bus usb@1c1a000 for devices... 1 USB Device(s) found
│   │    <> scanning bus usb@1c1a400 for devices... 1 USB Device(s) found
│   │    <> scanning bus usb@1c1b000 for devices... 1 USB Device(s) found
│   │    <> scanning bus usb@1c1b400 for devices... 1 USB Device(s) found
│   │    <> scanning bus usb@1c1c000 for devices... 1 USB Device(s) found
│   │    <> scanning bus usb@1c1c400 for devices... 1 USB Device(s) found
│   │    <> scanning bus usb@1c1d000 for devices... 1 USB Device(s) found
│   │    <> scanning bus usb@1c1d400 for devices... 1 USB Device(s) found
│   │    <>        scanning usb for storage devices... 0 Storage Device(s) found
│   │    <> Hit any key to stop autoboot:  0 
│   │    <> => 
│   ├─[rpi2] tee '/tmp/tbot-workdir/uboot-orangepipc/test/py/u_boot_boardenv_tbot_OrangePi PC.py' >/dev/null
│   ├─[rpi2] cd /tmp/tbot-workdir/uboot-orangepipc
│   ├─[rpi2] ./test/py/test.py --build-dir . --board-type 'tbot-OrangePi PC'
│   │    ## +u-boot-test-flash tbot-OrangePi PC na
│   │    ## ========================================================================= test session starts =========================================================================
│   │    ## platform linux -- Python 3.7.3, pytest-3.10.1, py-1.7.0, pluggy-0.8.0
│   │    ## rootdir: /tmp/tbot-workdir/uboot-orangepipc/test/py, inifile: pytest.ini
│   │    ## collected 327 items                                                                                                                                                   
│   ├─[rpi2] relay-card off
│   ├─POWEROFF (OrangePi PC)
│   ├─[rpi2] relay-card off
│   │    ## 
│   │    ## test/py/tests/test_000_version.py E                                                                                                                             [  0%]
│   │    ## test/py/tests/test_bind.py ss                                                                                                                                   [  0%]
│   │    ## test/py/tests/test_button.py s                                                                                                                                  [  0%]
│   │    ## test/py/tests/test_dfu.py s                                                                                                                                     [  0%]
│   │    ## test/py/tests/test_dm.py EE

Closing of SSH connection to board over Labhost from the board in interactive session lands you on the Labhost

I have this setup:

Lab:

import contextlib
import tbot
from tbot.machine import connector, linux


class StartBashInitializer(tbot.machine.Initializer):
    @contextlib.contextmanager
    def _init_machine(self):
        linux.util.wait_for_shell(self.ch)
        self.ch.sendline("bash --norc --noprofile")
        yield None


class Fuji(
    connector.ParamikoConnector,
    StartBashInitializer,
    linux.Bash,
    linux.Lab,
):
    hostname = "fuji.lan"
    username = "ch"

    @property
    def workdir(self):
        return linux.Workdir.static(self,
                                    f"/home/{self.username}/work/tbot-workdir")


def register_machines(ctx: tbot.Context) -> None:
    ctx.register(Fuji, tbot.role.LabHost)

Board:

import tbot
import contextlib
import time

from tbot.machine import connector, linux


class BoardLinux(connector.SSHConnector, linux.Bash):
    name = "board"
    hostname = "user"
    username = "root"

    @classmethod
    @contextlib.contextmanager
    def from_context(cls, ctx: tbot.Context):
        with tbot.ctx() as cx:
            lh = cx.request(tbot.role.LabHost)
            lo = cx.request(tbot.role.LocalHost)

            try:
                tbot.log.message("Power board on")
                lo.exec0("curl", "http://tasmota-0061.lan/cm?cmnd=Power%20On")

                tbot.log.message("Waiting for connection...")
                while not lh.test("timeout", "5", "ssh", BoardLinux.hostname, "true"):
                    tbot.log.message("still waiting ...")

                with cls(lh) as m:
                    yield m
            finally:
                tbot.log.message("Power board off")
                lo.exec0("curl", "http://tasmota-0061.lan/cm?cmnd=Power%20Off")


def register_machines(ctx: tbot.Context) -> None:
    ctx.register(BoardLinux, tbot.role.BoardLinux)

Starting a interactive session with the interactive_linux and closing the connection on the board via for instance a reboot command. Will drop me to the LabHost (fuji).

I use the current master commit: 0be8d7e of tbot

Better Verbosity management

The current verbosity levels do not provide much valuable control. I personally always have it set to Verbosity.STDOUT (= -vv) and I assume most other people have as well. Instead of traditional verbosity like it is implemented now, I think it makes more sense to more towards finer-grained control that is based on "event groups": As an example, let's look at commands:

  • E.g. for the uboot_build testcase, commands like make all or make defconfig are very important. They contain valuable information and show valuable output to the user.
  • On the other hand commands like mkdir -p $WORKDIR are generally uninteresting. They just provide visual noise and "leak" implementation details that most users don't care about.

These 'necessary but uninteresting' commands should be hidden by default while important commands should always be shown. Similarly, for some commands it is valuable to know they were executed but their output does not generally provide value. The Path.{write,read}_{text,binary}() family of functions is an example of that (in some situations).

kconfig_set_enabled misses to call olddefconfig

kconfig.enable can be used to enable a Kconfig option. But this is not enough. One has to run 'make olddefconfig' to avoid requests for input during the uboot_build target:

│   ├─Calling kconfig_set_enabled ...
│   │   ├─Enabling CONFIG_UNIT_TEST option ...
│   │   ├─[rpi2] sed -i '/^\(# \)\?CONFIG_UNIT_TEST\(=[ym]\| is not set\)$/cCONFIG_UNIT_TEST=y' /mnt/iscsi/user/workspace/tbotrunner/tbot-workdir/uboot-orangepipc/.config
│   │   └─Done. (0.019s)
│   ├─Calling uboot_make ...
│   │   ├─[rpi2] nproc --all
│   │   │    ## 4
│   │   ├─[rpi2] make -j 4 all
│   │   │    ## scripts/kconfig/conf  --syncconfig Kconfig
│   │   │    ## *
│   │   │    ## * Restart config...
│   │   │    ## *
│   │   │    ## *
│   │   │    ## * Unit tests
│   │   │    ## *
│   │   │    ## Unit tests (UNIT_TEST) [Y/n/?] y
│   │   │    ##   Unit tests for library functions (UT_LIB) [Y/n/?] (NEW) 

A good solution would be to call 'make olddefconfig' in UBootBuilder._build() after do_configure() by default.

Roadmap

This issue is an attempt to keep a list of features that should be implemented at some point. Individual features might get their own tracking issues if they are big enough.

Documentation Generation

A very nice feature of the original tbot was automatic documentation generation. We want this again, with new tbot but the exact API still needs to be defined.


Completed

"Long running"/Interactive Commands

This feature has hit master in commit 529f00c ("linux-shell: Implement interactive commands")! Documentation is here: LinuxShell.run()

More Path APIs

This feature was implemented in commits 29d5c0a ("linux.path: Add write_text and read_text methods") and 816f664 ("linux.path: Add write_bytes and read_bytes methods")! Documentation:

U-Boot test.py integration

This feature was added with commit 1fef92b ("tc.uboot: Add integration with U-Boot's test/py").

Deterministic CWD for machines

Currently, the CWD of a machine is largely undefined: For SubprocessConnecter it is going to be the CWD of tbot itself, for SSHConnector and similar, it is most likely $HOME. I think it would make more sense to always cd to a machine's workdir during initialization to provide a deterministic baseline.

Memory leak in SubprocessChannelIO

Lately I did some experiments with tbot powercycling a DUT via SSH which worked like a charm at first but kept failing after a few hundred cycles. After some investigation I found that this seems to occur since tbot does not properly close some file-descriptors.
To simplify the problem (and hopefully make it reproducible) I ran tbot with following setup:

config/eval.py:

from tbot import role
from tbot.machine import board, connector

class EvalBoard(connector.SubprocessConnector, board.Board):
    pass

def register_machines(ctx):
    ctx.register(EvalBoard, role.Board)

tc/test_fd.py:

import os
import tbot
import time

from tbot import role

@tbot.testcase
def test_fds():
    print(os.getpid())
    time.sleep(5)

    for i in range(2000):
        with tbot.ctx.request(role.Board) as b:
            tbot.log.message(f"RUN {i}")
            b.exec("echo", "Hello tbot!")

I execute the setup without any additional configuration like this: newbot -c config.eval tc.test_fd.test_fds
At some point in time (I guess this depends on your machine) this fails with an error message (I saw different kinds of errors, depending on wether I use SubprocessConnector or SSHConnector) like this:

│   ├─RUN 1015
│   ├─RUN 1016
│   └─Fail. (22.903s)
├─Exception:
│   Traceback (most recent call last):
│     File "/home/mebel/Environments/envtb/lib/python3.8/site-packages/tbot/machine/machine.py", line 158, in __enter__
│       self.ch = self._cx.enter_context(self._connect())
│     File "/home/mebel/Environments/envtb/lib/python3.8/site-packages/tbot/machine/connector/common.py", line 62, in _connect
│       return channel.SubprocessChannel()
│     File "/home/mebel/Environments/envtb/lib/python3.8/site-packages/tbot/machine/channel/subprocess.py", line 142, in __init__
│       super().__init__(SubprocessChannelIO())
│     File "/home/mebel/Environments/envtb/lib/python3.8/site-packages/tbot/machine/channel/subprocess.py", line 41, in __init__
│       self.p = subprocess.Popen(
│     File "/home/mebel/Environments/envtb/lib/python3.8/subprocess.py", line 858, in __init__
│       self._execute_child(args, executable, preexec_fn, close_fds,
│     File "/home/mebel/Environments/envtb/lib/python3.8/subprocess.py", line 1605, in _execute_child
│       errpipe_read, errpipe_write = os.pipe()
│   OSError: [Errno 24] Too many open files
├─────────────────────────────────────────
└─FAILURE (22.936s)

I modified the test like this to "freeze" execution at some point in time:

@tbot.testcase
def test_fds():
    print(os.getpid())
    time.sleep(5)

    for i in range(2000):
        with tbot.ctx.request(role.Board) as b:
            tbot.log.message(f"RUN {i}")
            b.exec("echo", "Hello tbot!")
            if i >= 5:
                b.interactive()

I get this output:

tbot starting ...
├─Calling test_fds ...
61006
│   ├─RUN 0
│   ├─RUN 1
│   ├─RUN 2
│   ├─RUN 3
│   ├─RUN 4
│   ├─RUN 5
│   ├─Entering interactive shell...
│   ├─Press CTRL+] three times within 1 second to exit.
bash-5.0$ Hello tbot!
bash-5.0$

When I inspect the open file-descriptors belonging to the tbot process with lsof -p 61006 I see the following:

...
newbot  61006 mebel    0u   CHR  136,6      0t0        9 /dev/pts/6
newbot  61006 mebel    1u   CHR  136,6      0t0        9 /dev/pts/6
newbot  61006 mebel    2u   CHR  136,6      0t0        9 /dev/pts/6
newbot  61006 mebel    3r   CHR    1,9      0t0       10 /dev/urandom
newbot  61006 mebel    4u   CHR    5,2      0t0       88 /dev/ptmx
newbot  61006 mebel    5u   CHR 136,10      0t0       13 /dev/pts/10 (deleted)
newbot  61006 mebel    6u   CHR 136,11      0t0       14 /dev/pts/11 (deleted)
newbot  61006 mebel    7u   CHR 136,12      0t0       15 /dev/pts/12 (deleted)
newbot  61006 mebel    8u   CHR 136,13      0t0       16 /dev/pts/13 (deleted)
newbot  61006 mebel    9u   CHR 136,14      0t0       17 /dev/pts/14 (deleted)
newbot  61006 mebel   10u   CHR 136,15      0t0       18 /dev/pts/15

It seems like the pseudo-terminals are closed properly (since they get deleted) but the file-descriptors remain open.
@Rahix I already found a fix for the described behavior and will happily supply a PR, but could you first try to reproduce my issue and confirm it as a bug? Maybe even I am doing something wrong or unintended with my test as well?

Weird power cycle recursion

Hey!

I'm trying to test u-boot using tbot. I followed the documentation and created, on my config, a generic Board for running picocom connection and handling power cycling:

class OrangePi(
    connector.ConsoleConnector,
    board.PowerControl,
    board.Board
):
    baudrate = 115200
    serial_port = "/dev/ttyUSB0"

    def poweron(self):
        with tbot.ctx.request(tbot.role.BoardUBoot) as ub:
            ub.exec0("reset")

    def poweroff(self):
        pass

    def connect(self, mach):
        return mach.open_channel("picocom", "-b", str(self.baudrate), self.serial_port)

And a UBootShell for handling just u-boot stuff:

class OrangePiUBoot(
    board.Connector,
    board.UBootAutobootIntercept,
    board.UBootShell
):
    prompt = "=> "

They are both registered:

def register_machines(ctx):
    ctx.register(OrangePi, tbot.role.Board)
    ctx.register(OrangePiUBoot, tbot.role.BoardUBoot)

My testcase should just run dhcp for now:

import tbot

@tbot.testcase
def test_uboot_dhcp() -> None:
    with tbot.ctx.request(tbot.role.BoardUBoot) as ub:
        ub.exec0("dhcp")

However, when I run that, I see a lot of parallel powerons running:

newbot -c config.orange_pi_test_config tc.interactive.test_uboot_dhcp
tbot starting ...
├─Calling test_uboot_dhcp ...
│   ├─[local] picocom -b 115200 /dev/ttyUSB0
│   ├─POWERON (orange-pi)
│   ├─[local] picocom -b 115200 /dev/ttyUSB0
│   ├─POWERON (orange-pi)
│   ├─[local] picocom -b 115200 /dev/ttyUSB0
│   ├─POWERON (orange-pi)
│   ├─[local] picocom -b 115200 /dev/ttyUSB0
│   ├─POWERON (orange-pi)
│   ├─[local] picocom -b 115200 /dev/ttyUSB0
│   ├─POWERON (orange-pi)
│   ├─[local] picocom -b 115200 /dev/ttyUSB0
RecursionError: maximum recursion depth exceeded while calling a Python object

Then it fails with max recursion depth.

Am I missing something? The only way I know that I can powercycle on u-boot is by resetting, so I did it according to the documentation (poweron handles the reset while poweroff just passes).

Add LinuxAlwaysOnMachine

Currently, TBot will always turn the board off after running. For boards with a long boot cycle, this can considerably slow down test development. To allow quicker iterations, there should be some kind of LinuxAlwaysOnMachine that allows accessing an already running board.

Continuously print console to stdout

Hello,

We have important kernel messages printed to the device console during tests and would like to have them printed continuously while the tests are running. These messages are needed for debugging when the tests fail.

Currently, to have the console's recent messages printed out, we need to send a command using exec0() method (our device is configured to use connector.ConsoleConnector).

Is there a way in TBot to have the console continuously print the console to stdout? I thought about having a thread that every second polls the channel for new data and printing it out using channel.read_until_timeout(1) method.

Thanks,
Yaniv

PS: Thank you for adding the utils.copy_to_dir() functionality :) (was out on vacation and returned today).

Timeout error while trying to reach the login prompt of the serial port over Paramiko Labhost

It appears that there is an error while trying to establish a connection to a serial port using Paramiko through tbot. The issue arises when encountering a Paramiko timeout reading while trying to connect to the login prompt. The code is designed to set up connections to a LabHost, a board, and a board's Linux environment. However, it fails to reach the expected login prompt within the specified timeout

import time
import tbot
import contextlib
import json
from tbot.machine import connector, board, linux
from tbot.machine.linux.linux_shell import LinuxShell

class StartBashInitializer(tbot.machine.Initializer):
    @contextlib.contextmanager
    def _init_machine(self):
        linux.util.wait_for_shell(self.ch)
        self.ch.sendline("bash --norc --noprofile")
        yield None

class RemoteHost(
    connector.ParamikoConnector,
    StartBashInitializer,
    linux.Bash,
):
    with open('config/config.json', 'r') as file:
        data = json.load(file)
        hostname = data['LabHost']['ip_address']
        username = data['LabHost']['username']
        authenticator = linux.auth.PasswordAuthenticator(data['LabHost']['password'])

class Device(connector.ConsoleConnector, board.Board):
        with open('config/config.json', 'r') as file:
               data = json.load(file)
               baudrate = data['DUT']['baudrate']
               serial_port = data['DUT']['serial_port']
               file.close()
        tbot.log.message("Waiting for serial logging...") 
        def connect(self, mach):
             return mach.open_channel("picocom", "-b", str(Device.baudrate), Device.serial_port)

class DeviceLinux(
    board.Connector,
    board.LinuxBootLogin,
    linux.Ash
):  
    username = "root"
    password = None
    # password_prompt=None
    # askfirst_prompt= 'Please press Enter to activate this console.'
    # login_prompt =  tbot.Re(r"~ # .{0,100}")
    login_prompt = "~#"
    # login_delay=5
    # no_password_timeout = None
    boot_timeout = 5

def register_machines(ctx):
    ctx.register(RemoteHost, tbot.role.LabHost)
    ctx.register(DeviceLinux, tbot.role.BoardLinux)
    ctx.register(Device, tbot.role.Board)

shell.copy() does not accept globs in source path

Hello,

When trying to copy multiple files filtered via glob to the device, shell.copy() fails with the following error:

tests/some_test.py:183: in copy_file
    shell.copy(*sp.glob('*'), dp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (Path(<tbot.selectable.LocalLabHost object at 0x7f5de90249f0>, '/home/ubuntu/tbot/tests/data/file1'), Path(<tbot.selec....py'), Path(<tbot.selectable.LocalLabHost object at 0x7f5de90249f0>, '/home/ubuntu/tbot/tests/data/file2'), ...)
kwargs = {}

    @functools.wraps(tc)
    def wrapped(*args: typing.Any, **kwargs: typing.Any) -> typing.Any:
        with tbot.testcase(tc.__name__):
>           return tc(*args, **kwargs)
E           TypeError: copy() takes 2 positional arguments but 7 were given

venv/lib/python3.8/site-packages/tbot/decorators.py:62: TypeError

Below is the code we use for copying files:

    with tbot.ctx.request(tbot.role.LabHost) as lh, \
        tbot.ctx.request(tbot.role.BoardLinux) as lnx:

        sp = linux.Path(lh, '/home/ubuntu/tbot/tests/data')
        dp = linux.Path(lnx, '/home/root')
        shell.copy(*sp.glob('*'), dp)

The current workaround is to get the file list from the glob and using a loop to copy the files:

    with tbot.ctx.request(tbot.role.LabHost) as lh, \
        tbot.ctx.request(tbot.role.BoardLinux) as lnx:

        sp = linux.Path(lh, '/home/ubuntu/tbot/tests/data')
        dp = linux.Path(lnx, '/home/root')
        files = sp.glob('*')
        for f in files:
             shell.copy(f, dp)

It would be great to have shell.copy() support copying files with glob.

thanks,
Yaniv

Send arbitrary commands via uboot console

Is there any way to send the Enter key as an arbitrary command via u-boot console?
My purpose here is to make the md increment the next result address and read it back.

Unhandled escape sequences clobber tbot output

I've seen a shell send the following escape sequences which tbot doesn't handle well:

│   ├─(690600)< '\x1b7\x1b[r\x1b[999;999H\x1b[6n'

Individually, these mean:

Sequence Meaning
\x1b7 Save Cursor (DECSC)
\x1b[r ?
\x1b[999;999H Cursor Position [row;column] (default = [1,1]) (CUP)
\x1b[6n Report Cursor Position (CPR) [row;column]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.