clarkwang / sexpect

Expect for Shells

What's sexpect

sexpect is another implementation of Expect which is specifically designed for Shell scripts (sh, bash, ksh, zsh, ...).

sexpect is designed in the client/server model. sexpect spawn PROGRAM [option].. starts the server (running as a daemon) and runs the specified PROGRAM in background. Other sexpect sub-commands (send, expect, wait, ...) communicate to the server as client commands.

How to build

Sexpect uses CMake for building.

$ cd /path/to/cloned/sexpect/
$ mkdir build
$ cd build
$ cmake ..
$ make
$ make install

By default it will install to /usr/local/ and sexpect will be installed to /usr/local/bin/. To change the installation location, run cmake like this:

$ cmake -D CMAKE_INSTALL_PREFIX=/opt/sexpect  ..

Supported platforms

Tested on:

• OpenWRT 15.05.1, ramips/mt7620 (on Newifi Mini, or Lenovo Y1 v1)
• Debian Linux 9 (Stretch)
• macOS 10.13 (High Sierra)
• FreeBSD 11.1
• Cygwin on Windows 10

Manual

See doc/sexpect.adoc.

Examples

See the examples/ dir.

sexpect's Issues

`chk -err 0 -is eof` fails.

$ sexpect ver
sexpect 2.1.11
$ sexpect chk -errno 0 -is eof
[ERROR] both -errno and -is must be specified
$

[v2.3.4] issue with "-zombie-idle"

The following scenario is supposed to print the defunct sleep process but with v2.3.4 it printed nothing.

sexpect sp -z-idle 5 sleep 5; pid=$( sexpect get -pid ); sleep 6; ps p $pid

Feature request - tag or/and release for new version

Hi,
I'm trying to create sexpect package for OpenWrt and it would be useful if there was tag or release (that would be even better).for specific version.

[v2.3.12] Add an option to pass a file descriptor to the "send" command

Inspired by this so named pipes could be used.

Passing it to -file returns:

[ERROR] not a regular file: /dev/fd/63

Not sure if it can be combined or split into its own arg like -file-fd.

[v2.3.11] New subcommand "get -expect-buf N": dump data in the internal expect buffer

... for easy debugging when writing sexpect scripts.

syntax error: unexpected "("

I manage to cross compile for openwrt successfully
To compile, execute cmake below
cmake .. -D CMAKE_CXX_COMPILER=mipsel-openwrt-linux-musl-g++ -D CMAKE_INSTALL_PREFIX=/usr/bin

However got error below while execute binary
root@PRIUS:~# sexpect --help
/usr/bin/sexpect: line 1: syntax error: unexpected "("

Anything missing?

The option "-zombie-ttl N" does not work as expected

See the following example:

$ cat zombie.sh
export SEXPECT_SOCKFILE=/tmp/zombie.$$.sock
sexpect spawn -zttl 5 sleep 10
sleep 10
sexpect wait
$ bash zombie.sh
[ERROR] connect: No such file or directory (2)
$

[v2.3.13] interact -subst PATTERN::REPLACE

interact <-subst|-sub> PATTERN::REPLACE

• -nocase would cause PATTERN to be case-insensitive
• -cstring impacts both PATTERN and REPLACE
• REPLACE can include (0), (1), ..., (9) for RE match groups
  • For example, -subst "(foo)(bar)::(0) (1) (2)" would change foobar to foobar foo bar.
• -subst can be specified for at most 10 times.

[doc] get -expect-buf

• sexpect -h
• man sexpect

[v2.3.8] 0.4% CPU usage on macOS 11.6 (32GB Mem)

On macOS 11.6 Big Sur (32GB Mem), a background sexpect for ssh -D (with or without -v) takes 0.4% CPU which is unnecessarily high.

Build size

Not an issue but a question. How much flash does the built binary consume on the architectures that you have built on? Looking for a lightweight expect replacement on a pretty flash constrained router.

expect -re can't match specific character ~ $ cygwin on windows 10

expect -re can't match specific character ~ $
cygwin on windows 10

updated example/ssh.sh
expect -re ':~ \$ $'  try to match prompt before run any command
but failed with  'ret=207'

expect -ex 'raspberrypi:'   works
expect -re 'raspberrypi:'   works
expect -ex 'raspberrypi:~'   don't work
expect -re 'raspberrypi:~'   don't work

cygcheck (cygwin) 3.1.2  X86-64
Windows 10 Professional Ver 10.0 Build 18363   X86-64

root@DESKTOP-XYZ ~/sexpect-master/examples
$ bash -x ./ssh.sh -p raspberry [email protected]

+ sexpect expect -nocase -re 'raspberrypi:~'

Linux raspberrypi 4.19.75+ #1270 Tue Sep 24 18:38:54 BST 2019 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

while true; do
    sexpect expect -nocase -re 'raspberrypi:~'
	ret=$?
	if [[ $ret == 0 ]]; then
	    break
    elif sexpect chkerr -errno $ret -is eof; then
        sexpect wait
        exit
    elif sexpect chkerr -errno $ret -is timeout; then
        sexpect close
        sexpect wait
        fatal "timeout waiting for password prompt"
    else
        fatal "unknown error: $ret"
    fi
done

sexpect send -enter 'uptime'

sexpect set -idle 5
sexpect interact

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Jan 25 04:40:09 2020 from 192.168.1.8

pi@raspberrypi:~ $ + ret=207
+ [[ 207 == 0 ]]
+ sexpect chkerr -errno 207 -is eof
+ sexpect chkerr -errno 207 -is timeout
+ sexpect close
+ sexpect wait
+ fatal 'timeout waiting for password prompt'
+ echo '!! timeout waiting for password prompt'
!! timeout waiting for password prompt
+ exit 1


$ cygcheck.exe -V
cygcheck (cygwin) 3.1.2
System Checker for Cygwin
Copyright (C) 1998 - 2019 Cygwin Authors

$ cygcheck.exe -s
Cygwin Configuration Diagnostics
Current System Time: Sat Jan 25 04:36:19 2020

Windows 10 Professional Ver 10.0 Build 18363

"-discard" does not work for `od -v /dev/zero`

(Issue was found when demo'ing to MJZ.)

To reproduce:

export SEXPECT_SOCKFILE=/tmp/discard.sock
sexpect sp -discard od -v /dev/zero
pid=$( sexpect get -pid )
st=$( ps p $pid ho state )
# $st should be `R' but its `S'
echo $st

cmake: add "make install" support

Refs:

• install()
• CMAKE_INSTALL_PREFIX
  • "This variable defaults to /usr/local on UNIX [...]"
• DESTDIR

[v2.3.14] add a new cli option to enable "regcomp(cflags|=REG_NEWLINE)"

Related subcommands

• expect -re PATTERN
• expect -glob PATTERN
• interact -re PATTERN
• interact -subst PATTERN::REPLACE

New option name?

• -REG_NEWLINE
• -anchor-newline | -anchor

Implementation

• With -anchor-newline, is it good idea to convert \r+\n to \n before calling regexec()?
  • Don't bother!

Don't run the specified command as session leader?

• There are interesting rules for session leaders.
• Run the command with sh -c? bash -c? Allow users to specify the shell?
  • Some shells (like Bash?) is too smart and they exec the last command.
    • Bash can be fooled by bash -c 'cmd ...; exit'.

Add testcases for v2.3.12..v2.3.14

Release missing tags

It seems the last one was 2.3.8, but current is 2.3.11?

[v2.3.9] Add version check when the client connects to the server

New client behavior

• Client sends its VERSION in the HELLO message to server.
  • Old sever ^(v2.3.8-) ignores the VERSION field in the HELLO message.
• Client quits the session if there's no VERSION in server's HELLO message.
• Client quits the session if its VERSION is not the same as server.

New server behavior

• Server refuses the connection if client's HELLO message has no VERSION info.
• Server refuses the connection if its VERSION is not the same as client.
• Server sends its VERSION back to client in the HELLO message.

Package sexpect in Debian

May need to update CMakeLists.txt and add make install support.

"get" commands like "spawn get -tty" should not output the "-zttl" value

$ sexpect version
sexpect 2.3.2
$
$ export SEXPECT_SOCKFILE=/root/tmp/sexpect-pniYll.sock
$ sexpect spawn -zttl 1234 sleep 60
$ sexpect get -tty
/dev/pts/2
1234        <-- The -zttl value should not be outputed here.
$

Implement "interact_out" like in Expect?

Issue #19 added interact -re PATTERN but how to get interact_out?

For now, the interact -re PATTERN logic is implemented in client side. Moving it to server side will complicate the server side code quite a lot. And interact must exit with different codes to distinguish between

1. EOF
2. ctrl-]
3. -re PATTERN
4. Other errors.

[usability] send -file FILE: add new option "-strip" to remove blank chars (space, tab, \r, \n)

Would be helpful when the file contains only a password (e.g. if the password is followed by a few extra invisible white spaces).

• -lstrip (left side)
• -rstrip (right side)
• -strip (both)

How to determine the moment when spawned process requests stdin?

Hi. Nice work 👍.

I am looking for a way to colorize bash --verbose output. In other words my task is to execute a bash script highlighting every command before execution. During hours I looked for a solution but nothing worked as expected. Expected is:

$ command  # highlighted
out line 1
out line 2
$ multi-
  line-
  command2   # highlighted
out2
$ cmd3  # highlighted
...

Currently I think the best solution would be is to run background bash process and send commands to it when it is ready to read them. Parent script will read script to execute line by line and write colorized line to stderr, then send them to spawned sub-bash when it is wait for new input. This is required to do not mix outputs.

So the question

How to determine the moment when spawned process requests stdin with sexpect?

see my question on SO https://stackoverflow.com/questions/52708607/how-to-get-know-if-a-process-is-waiting-on-stdin

padavan/entware compile success,but bin doesnt work

padavan/entware, linux version 3.4, compile success, by modify Makefile install directory from “/usr/bin” to “/opt/bin”, ect……
But when putting ipk file to padavan system, and installing with opkg successfully, accessing this tool from shell is not O.K. as it should be.
Error occurs……
Dont know where the problem is.……

[v2.3.10] "-nonblock on": don't drop data too quickly

Currently with v2.3.8:

$ sexpect sp -nb bash --norc
$ sexpect i
        <-- Here you cannot see the bash prompt because, with nonblock=on, all data are marked
        <-- as old. You have to use `sexpect i -lb' to see recent old data.

With v2.3.9, it marks data as old only when the internal buffer is full so sexpect i in above example can see the bash prompt.

$ sexpect sp -nb bash --norc
$ sexpect i
bash-5.2$ 

[v2.3.7-dev] 2 issues with "interact -re"

Example:

$ sexpect sp bash --norc
$ sexpect i -re '.*'
bash-5.1# [BUG!?] cli_disconn: not supposed to receive tag TAG_MATCHED
$ sexpect i -re xxx

bash-5.1# exit
exit                <== it hangs here. even ctrl-c does not work.

option to discard color codes & al

Some programs (like bluetoothctl) report messages with color codes that can't be disabled: that makes sexpect regex quite challenging to craft (expect seems immune to that).
Would it make sense to have a mode/option to discard all of such formatting for regex/exact/glob matches?
Thanks for consideration.

[v2.3.7] interact -re PATTERN: detach upon matching a specified pattern

• I used to think this is not quite useful. Changed my idea recently when writing a script which needs to input password in an interactive way for security's sake.
• The whole logic can be done in the client side. Don't complicate the server side.

A bit more background:
In a corporate env it's not appropriate to ask users to pass their corporate account's passwords in command line. After spawning a program, when it prompts for the password, enter interactive mode and ask users to input their password manually, then automatically exit interactive mode when matching a specified pattern, then continue running the rest of the script.

Add test for "-zombie-idle N"

See issue #16.

[Usability] Server side: auto wait the child and exit 30min after pty/slave closes and child exits

This would help avoid too many zombies running around.

New option:

• spawn -zombie-ttl N (N=-1 for infinity; Default: 10min? 30min?)
• get -zombie-ttl? Not very necessary. get -all should be enough.
• set -zombie-ttl? Don't bother.

Add more detailed debug messages (like "expect -d")