pycontribs / subprocess-tee Goto Github PK

A subprocess.run drop-in replacement that supports a tee mode, being able to display output in real time while still capturing it. No dependencies needed

License: MIT License

Python 95.84% Dockerfile 4.16%

subprocess tee pep-621

subprocess-tee's People

Contributors

Stargazers

Watchers

Forkers

stoned apatard ssbarnea onlyabhi52 doytsujin alvistack isabella232 dvzrv andrey1908 gotmax23 kotlinisland johanneskastl robertschweizer jonashaag

subprocess-tee's Issues

(🐞) Doesn't work on Windows

from subprocess import run
from subprocess_tee import run as run_tee

run('notepad.exe test.py')  # works
run_tee('notepad.exe test.py')  # FileNotFoundError: [WinError 3] The system cannot find the path specified

RuntimeError: There is no current event loop in thread 'MainThread'.

I'm attempting to use subprocess tee in a simple, synchronous application, but when I run it in this code, it fails with the following traceback:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.3/x64/lib/python3.12/site-packages/safety-tox.py", line 54, in <module>
    __name__ == '__main__' and run(sys.argv[1:])
                               ^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.3/x64/lib/python3.12/site-packages/safety-tox.py", line 51, in run
    raise SystemExit(Handler().run(args))
                     ^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.3/x64/lib/python3.12/site-packages/safety-tox.py", line 33, in run
    proc = self.runner(cmd)
           ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.3/x64/lib/python3.12/site-packages/subprocess_tee/__init__.py", line 138, in run
    loop = asyncio.get_event_loop_policy().get_event_loop()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.3/x64/lib/python3.12/asyncio/events.py", line 676, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'MainThread'.

I'm using Python 3.12 alpha because that's the environment where safety-tox is needed. I can't replicate the issue on Python 3.11.

Here's a reproducer without safety-tox:

 ~ $ py -3.12 -m pip-run subprocess-tee -- -c "from subprocess_tee import run; run(['echo', 'foo'])"
Collecting subprocess-tee
  Using cached subprocess_tee-0.4.0-py3-none-any.whl (5.1 kB)
Installing collected packages: subprocess-tee
Successfully installed subprocess-tee-0.4.0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/var/folders/sx/n5gkrgfx6zd91ymxr2sr9wvw00n8zm/T/pip-run-hroxhf41/subprocess_tee/__init__.py", line 138, in run
    loop = asyncio.get_event_loop_policy().get_event_loop()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/events.py", line 676, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'MainThread'.

Possibly this issue should be reported upstream with CPython?

Does not print partial lines

I thought subprocess-tee would fit in nicely for the following example

from subprocess_tee import run
o = run("ssh remotehost -t sudo cat /super/secret/file"))
# now trim of the password prompt from o

i.e. someone running my python script should see the sudo password prompt from the remote host, and I can capture the cat output in a local variable (i don't want my users to blindly type their sudo passwords, assuming the ssh connection got established, though they still have to trust me that i'm not doing anything nasty with the password). yet, this doesn't work as desired because there is no linefeed after [sudo] password for <user>: and the prompt only becomes printed by subprocess-tee after the user hits return.

(🐞) There are no types for the kwargs

The run from subprocess is fully typed, could we see the same thing here?

Looking for co-maintainers

While this is a very small library that does not need much maintenance I am interested in having a couple of co-maintainers, so in case one of us becomes unavailable, users would not end-up having to fork it to patch it.

Passing cwd argument is ignored

At this moment passing cwd to run() does not change the current working directory.

$ python                                                                                                                        [12:16:00]
>>> from subprocess_tee import run
>>> run("pwd")
/Users/ssbarnea/c/a/molecule/src/molecule/test/scenarios/driver/delegated
CompletedProcess(args='pwd', returncode=0, stdout='/Users/ssbarnea/c/a/molecule/src/molecule/test/scenarios/driver/delegated\n', stderr='')
>>> run("pwd", cwd="/")
/Users/ssbarnea/c/a/molecule/src/molecule/test/scenarios/driver/delegated
CompletedProcess(args='pwd', returncode=0, stdout='/Users/ssbarnea/c/a/molecule/src/molecule/test/scenarios/driver/delegated\n', stderr='')
>>>

Passing DEVNULL to stderr or stdout args should silence it

In order to match behavior of subprocess.run, passing subprocess.DEVNULL to stderr and/or stdout args, should silence the command.

At this moment this works only by passing quiet=True. A PR would be welcomed.

Windows: Argument list converted to string incorrectly

When passing a list of arguments, subprocess-tee converts that to a single string with shlex.join. If one of the arguments contains a backslash, it is enclosed in single-quotes ('). This leads to the following error on Windows:

$ python -c "from subprocess_tee import run; import sys; run([sys.executable, '-V'], executable=None)"
The filename, directory name, or volume label syntax is incorrect.

Workaround

Pass the arguments as string.

Fix

shlex.join is only meant for POSIX systems. On Windows, subprocess.list2cmdline can be used to convert argument lists to strings, but is not officially part of the Python API (https://bugs.python.org/issue10838).

Difference between this and `ubelt.cmd`, others? What makes this problem so difficult?

Greetings! 👋

I am trying to understand the lay of the land of cross-platform teeing of subprocess (py). I've also found ubelt.cmd (tests) by @Erotemic.

I originally got here through StackOverflow in a comment, but I can't find the post.

In either event, the problem comes up a lot:

I am racking my brain when I read the code / solutions for this issue and wondering:

What makes this issue (subprocess + tee) so tricky to solve? e.g. Is it the program's IO? The environment / system? asyncio?
What are these libraries, such as subprocess-tee, doing with the command?
Does ubelt.cmd do the same thing? Is it different n some way?
Is this the kind of thing a standard library PEP would be able to tackle in a cross-platform way?

Sorry for any interruption, as this is a non-code post and a bit meta 😄

If check=True, empty output becomes None

print(subprocess_tee.run(["cat", "/dev/null"]).stdout)
# Prints ""
print(subprocess_tee.run(["cat", "/dev/null"], check=True).stdout)
# Prints "None"

Cyclic dependency on `molecule`

In the nix package manager we're seeing a build failure in subprocess-tee because of the new molecule dependency in tests.

Just wondering if there is a special reason for why molecule is used in the test_func test? Would it be possible to change the test to use something like this?

subprocess.run(["echo", "hello world"], ...)

The problem is that molecule itself seems to depend on subprocess-tee so we're seeing a bit of a cyclic dependency unfortunately.

Also a quick question: it looks like the test is testing the Python built-in method subprocess.run. Could this test maybe be removed?

/bin/bash should not be hardcoded

Currently subprocess_tee assumes that /bin/bash is the default shell on non-windows platforms.

subprocess-tee/lib/subprocess_tee/__init__.py

Lines 24 to 30 in c7ef910

 async def _stream_subprocess(args, **kwargs) -> CompletedProcess: 

 if platform.system() == "Windows": 

 platform_settings: Dict[str, Any] = {"env": os.environ} 

 else: 

 platform_settings = {"executable": "/bin/bash"} 

 if "env" in kwargs: 

 platform_settings["env"] = kwargs["env"]

But there are a lot of platforms where bash is not the default shell and where it might not even be installed (lean Docker containers are a good example).

Some logic should be added for better detection of default shell and for handling of non-existent /bin/bash.

Related: ansible/molecule#2944

Run inside Jupyter

With the recent changes, subprocess-tee is using asyncio.run. However, this call tries to create a new loop, raising an exception:

RuntimeError: asyncio.run() cannot be called from a running event loop

To reproduce, just create a Jupyter notebook with this cell and run:

import subprocess_tee
subprocess_tee.run('sleep 1')

More details at jupyter blog or stackoverflow.

Windows: Multiple arguments unsupported

Single-argument commands work, e.g.

from subprocess_tee import run
run(["git"])

With multiple arguments, an error is raised:

$ python -c "from subprocess_tee import run; run(['git status'])"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python39\lib\site-packages\subprocess_tee\__init__.py", line 142, in run
    result = asyncio.run(_stream_subprocess(cmd, **kwargs))
  File "C:\Program Files\Python39\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Program Files\Python39\lib\asyncio\base_events.py", line 647, in run_until_complete
    return future.result()
  File "C:\Program Files\Python39\lib\site-packages\subprocess_tee\__init__.py", line 67, in _stream_subprocess
    process = await asyncio.create_subprocess_shell(
  File "C:\Program Files\Python39\lib\asyncio\subprocess.py", line 216, in create_subprocess_shell
    transport, protocol = await loop.subprocess_shell(
  File "C:\Program Files\Python39\lib\asyncio\base_events.py", line 1643, in subprocess_shell
    transport = await self._make_subprocess_transport(
  File "C:\Program Files\Python39\lib\asyncio\windows_events.py", line 3[94], in _make_subprocess_transport
    transp = _WindowsSubprocessTransport(self, protocol, args, shell,
  File "C:\Program Files\Python39\lib\asyncio\base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "C:\Program Files\Python39\lib\asyncio\windows_events.py", line 890, in _start
    self._proc = windows_utils.Popen(
  File "C:\Program Files\Python39\lib\asyncio\windows_utils.py", line 153, in __init__
    super().__init__(args, stdin=stdin_rfd, stdout=stdout_wfd,
  File "C:\Program Files\Python39\lib\subprocess.py", line [95]1, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Program Files\Python39\lib\subprocess.py", line 1420, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 3] The system cannot find the path specified

The reason is probably that platform_settings["executable"] is set on Windows: https://github.com/pycontribs/subprocess-tee/blob/v0.4.1/src/subprocess_tee/__init__.py#L57-L58

Workaround

Pass executable=None:

from subprocess_tee import run
run(["git", "status"], executable=None)

Allow redirecting stderr to stdout (stderr=sp.STDOUT)

Setting stderr=subprocess.STDOUT is currently ignored.

AttributeError: 'function' object has no attribute 'write'

subprocess_tee 0.3.1 introduces an attribute error that was not present in 0.2.0:

Task exception was never retrieved
future: <Task finished coro=<_read_stream() done, defined at /root/venv/lib/python3.7/site-packages/subprocess_tee/__init__.py:21> exception=AttributeError("'function' object has no attribute 'write'")>
Traceback (most recent call last):
  File "/root/venv/lib/python3.7/site-packages/subprocess_tee/__init__.py", line 25, in _read_stream
    callback(line)
  File "/root/venv/lib/python3.7/site-packages/subprocess_tee/__init__.py", line 86, in <lambda>
    _read_stream(process.stderr, lambda l: tee_func(l, err, stderr))
  File "/root/venv/lib/python3.7/site-packages/subprocess_tee/__init__.py", line 73, in tee_func
    print(line_str, file=pipe)
AttributeError: 'function' object has no attribute 'write'

Full molecule log: https://cirrus-ci.com/task/4613075711033344?logs=test#L247

Not work on windows

It seems that this package does not work on Windows.

(‼) subprocess-tee is not a drop-in replacement for subprocess

It uses shell by default:

from subprocess import run
from subprocee_tee import run as tee

tee("echo 1")  # 1
run("echo 1")  # FileNotFoundError: [Errno 2] No such file or directory: 'echo 1'

Either changes should be made to be consistent with subprocess, or the branding should be changed.

py38: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait()

Current implementation is producing a deprecation warning in py39:

subprocess_tee/__init__.py:46: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11.
    await asyncio.wait(

https://github.com/pycontribs/subprocess-tee/blob/main/lib/subprocess_tee/__init__.py#L46

If anyone knows how to address this deprecation warning without breaking backwards compatibility with py36+, I would greatly appreciate.

check=True isn't supported by run()

Currently, this can't be used as a drop-in replacement for subprocess.run() as passing check=True isn't supported directly. A simple fix would be to call check_returncode() manually on the returned CompletedProcess.

BTW, IMO subprocess_tee.run() should raise an error when it's passed unsupported arguments. This would avoid silent unexpected behavior.

How do you use this for try-catch exceptions?

Hey, i noticed this module doesn't support subprocess's CalledProcessError exception, ad the CalledProcess variable it has isn't an exception class (my compiler, pyright says so). How do I use this for try-catch code?
The code in question (Python 3)

def run_yay(cmd_args: str, pkg: str):
    """Runs yay
    Keyword Arguments:
    cmd_args    -- The arguments passed to pacman
    pkg         -- The packages to be installed
    """
    try:
        sb.run(
            ["yay %s %s" % (cmd_args, pkg)],
            stdout=sb.PIPE,
            stderr=sb.PIPE,
            shell=True,
            check=True,
        )
    except sb.CalledProcessError as e:
        logging.critical("Yay has run into an issue installing! Cannot continue!")
        logging.critical("Error Code: " + str(e.returncode))
        exit(1)
    else:
        logging.info(
            "Yay finished running with args: [" + cmd_args + "] and pkg: [" + pkg + "]"
        )

If check=False execution becomes async

If check=False is in the args then the run(..) function returns before execution has completed. I assume this is because CompletedProcess.returncode is a coroutine, so the calling thread doesn't block until it is accessed. When check=True this happens at the end of run(..).

Limitation because subprocess.run with timeout doesnt work for child processes

I am trying to use a timeout to kill a subprocess launched via .run(timeout=); this is not supported unlike the original subprocess.run()

To reproduce:

Python 3.8.12 (default, Oct 27 2021, 12:52:48) 
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from subprocess_tee import run
>>> run('sleep 10', shell=True, timeout=.1, capture_output=True)
CompletedProcess(args='sleep 10', returncode=0, stdout='', stderr='')

Whereas with subprocess.run the exception is correctly thrown:

Python 3.8.12 (default, Oct 27 2021, 12:52:48) 
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from subprocess import run
>>> run('sleep 10', shell=True, timeout=.1, capture_output=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/subprocess.py", line 495, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/local/lib/python3.8/subprocess.py", line 1028, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/local/lib/python3.8/subprocess.py", line 1869, in _communicate
    self._check_timeout(endtime, orig_timeout, stdout, stderr)
  File "/usr/local/lib/python3.8/subprocess.py", line 1072, in _check_timeout
    raise TimeoutExpired(
subprocess.TimeoutExpired: Command 'sleep 10' timed out after 0.1 seconds

The output should have been a TimedoutException.

Also tried

>>> run(['sleep','10'], shell=False, timeout=.1, capture_output=False)
CompletedProcess(args=['sleep', '10'], returncode=0, stdout='', stderr='')

not working on windows

maybe related to #15

On windows I'm trying to sublaunch a batch script (Which happens to be conda.bat). The equivalent works on unix/mac.

I had some success with: subprocess_tee.run(['…path to conda.bat', …], executable="c:/windows/system32/cmd.exe", …) but perhaps there's a better way.

the error I got is that windows cannot find the file specified.

test_rich_console_ex Test fail: duplicate values

On openSUSE Tumbleweed we are currently experiencing this test fail:

[  115s] =================================== FAILURES ===================================
[  115s] _____________________________ test_rich_console_ex _____________________________
[  115s] 
[  115s]     def test_rich_console_ex() -> None:
[  115s]         """Validate that ConsoleEx can capture output from print() calls."""
[  115s]         console = Console(record=True, redirect=True)
[  115s]         console.print("alpha")
[  115s]         print("beta")
[  115s]         sys.stdout.write("gamma\n")
[  115s]         sys.stderr.write("delta\n")
[  115s]         # While not supposed to happen we want to be sure that this will not raise
[  115s]         # an exception. Some libraries may still sometimes send bytes to the
[  115s]         # streams, notable example being click.
[  115s]         # sys.stdout.write(b"epsilon\n")  # type: ignore
[  115s]         proc = run("echo 123")
[  115s]         assert proc.stdout == "123\n"
[  115s]         text = console.export_text()
[  115s] >       assert text == "alpha\nbeta\ngamma\ndelta\n123\n"
[  115s] E       AssertionError: assert 'alpha\nbeta\...a\n123\n123\n' == 'alpha\nbeta\...ndelta\n123\n'
[  115s] E           alpha
[  115s] E           beta
[  115s] E         + beta
[  115s] E         + gamma
[  115s] E           gamma
[  115s] E           delta
[  115s] E         + delta...
[  115s] E         
[  115s] E         ...Full output truncated (3 lines hidden), use '-vv' to show
[  115s] 
[  115s] src/subprocess_tee/test/test_rich.py:23: AssertionError
[  115s] ----------------------------- Captured stdout call -----------------------------
[  115s] alpha
[  115s] beta
[  115s] gamma
[  115s] delta
[  115s] 123

Complete log (which also includes versions of all installed libraries):
log.txt

Do you know what the issue might be?

invalid pytest minimum version in setup.cfg

Hi,

I'm trying to create an rpm for this project as it's a newly added dependency of molecule, and while running the tests I got this error:

/+ /usr/bin/python3 -m pytest --version
pytest 6.0.2
ERROR: usage: main.py [options] [file_or_dir] [file_or_dir] [...]
main.py: error: unrecognized arguments: --durations-min=3

I checked and it seems like the agument --duration-min was only added on version 6.1.0 of pytest (https://docs.pytest.org/en/stable/changelog.html#features), while in the setup.cfg the min version of pytest is 5.4.0

subprocess-tee/setup.cfg

Line 73 in 4c38a99

pytest>=5.4.0

0.3.3 critical regression, no output displayed

I just yanked 0.3.3 release due to fatal flaw found on it, where it prevents molecule from displaying the output of executed commands.

Likely this was introduced by #44

This bug should track changes being done for fixing and preventing similar regressions in the future.

(🐞) Doesn't work from async code

from asyncio import run

from subprocess_tee import run as tee


async def foo():
    tee(['notepad.exe', 'test2.py'], shell=False)  # RuntimeError: asyncio.run() cannot be called from a running event loop
run(foo())

Until this is resolved it should be documented as a limitation.

Tests are installed

Hi! I'm packaging this project for Arch Linux.
I noticed that the tests are being installed alongside the files of this project. However, they are not required during runtime and should not be installed to the system.

Platform requirements not clearly advertised

I started using this project intending to use it to wrap tox to detect known failed builds, and I didn't realize that it only runs on some systems. Best I can tell, the non-Windows support is only declared in closed issues. Perhaps the project could expose the supported platforms is Trove classifiers so they appear prominently on the PyPI page.

Execution is always as a shell

When there is no space in the command

subprocess.run(["a"])

Will not use the shell, but with:

subprocess_tee.run(["a"])

will be run as a shell.

I think it would be fairly simple to update the functionality to use create_subprocess_shell if a string is passed, and create_subprocess_exec if a iterable is passed.

	async def _stream_subprocess(args, **kwargs) -> CompletedProcess:
	if platform.system() == "Windows":
	platform_settings: Dict[str, Any] = {"env": os.environ}
	else:
	platform_settings = {"executable": "/bin/bash"}
	if "env" in kwargs:
	platform_settings["env"] = kwargs["env"]

pycontribs / subprocess-tee Goto Github PK

subprocess-tee's People

Contributors

Stargazers

Watchers

Forkers

subprocess-tee's Issues

Workaround

Fix

Workaround

Recommend Projects

Recommend Topics

Recommend Org

Jobs