Flaky tests about oil

[andychu]

NOTE: test/spec.sh builtins often fails, because of the pwd -P case in dash. Not sure why this is. Is dash flaky?

This even happens when not running in parallel.

[Yorwba]

The dash issue is because another test creates the symlink without cleaning it up. When the test is run for dash, it can't create the symlink but doesn't abort, so afterwards the symlink is cleaned up. Then the test passes for the other shells.

I had already noticed and fixed this bug when I was working on adapting the spec tests, but forgot to separate it from the other changes. Sorry about that.

Honestly spec tests should run with a fresh, empty TMP directory every time, otherwise you get weird interference between tests like this.

[andychu]

Great catch! I gave each test file its own $TMP. And I also adjusted the test cases so they are less likely to clobber each other.

test/spec.sh alias fails for me on ubuntu 16.04, all tests fail for osh.

[andychu]

Can you paste the output? Right now on master it's "passing" for me, with 10 OSH failures and 1 success:

If the exit code is 0, that's considered OK. That means all the failures are "allowed".

andy@andy-VirtualBox ~/git/oilshell/oil (master)$ test/spec.sh alias
alias.test.sh
case    line    dash    bash    mksh    zsh     osh     osh_ALT
  0       9     pass    pass    pass    pass    FAIL    FAIL    basic alias
  1      19     pass    pass    pass    pass    FAIL    FAIL    alias with trailing space causes second alias expansion
  2      36     pass    pass    pass    pass    FAIL    FAIL    iterative alias expansion of first word
  3      47     pass    pass    pass    pass    FAIL    FAIL    expansion of alias with value
  4      58     BUG     pass    pass    pass    FAIL    FAIL    first and second word are the same
  5      70     pass    pass    pass    pass    FAIL    FAIL    first and second word are the same with trailing space
  6      79     pass    pass    pass    pass    FAIL    FAIL    defining multiple aliases, then unalias
  7      97     pass    pass    pass    pass    pass    pass    Invalid syntax of alias
  8     103     pass    pass    pass    pass    FAIL    FAIL    Dynamic alias definition
  9     112     pass    pass    pass    pass    FAIL    FAIL    Alias detection happens before expansion
 10     124     pass    pass    pass    pass    FAIL    FAIL    Alias name with punctuation
45 passed, 0 ok, 0 known unimplemented, 1 known bugs, 10 failed, 0 skipped
note: Got 10 allowed osh failures (exit with code 0)

My output:

ricardo@home:~/src/oil$ ./test/spec.sh alias
alias.test.sh
case	line	dash	bash	mksh	zsh	osh	
  0	  9	pass	pass	pass	pass	FAIL	basic alias
  1	 19	pass	pass	pass	pass	FAIL	alias with trailing space causes second alias expansion
  2	 36	pass	pass	pass	pass	FAIL	iterative alias expansion of first word
  3	 47	pass	pass	pass	pass	FAIL	expansion of alias with value
  4	 58	BUG	pass	pass	pass	FAIL	first and second word are the same
  5	 70	pass	pass	pass	pass	FAIL	first and second word are the same with trailing space
  6	 79	pass	pass	pass	pass	FAIL	defining multiple aliases, then unalias
  7	 97	pass	pass	pass	pass	FAIL	Invalid syntax of alias
  8	103	pass	pass	pass	pass	FAIL	Dynamic alias definition
  9	112	FAIL	FAIL	FAIL	FAIL	FAIL	Alias detection happens before expansion
 10	124	pass	pass	pass	pass	FAIL	Alias name with punctuation
39 passed, 0 ok, 0 known unimplemented, 1 known bugs, 15 failed, 0 skipped

FATAL: Got 15 failures (11 osh failures), but 10 are allowed

[andychu]

@granttrec Can you run this and paste this output?

test/spec.sh alias --range 9 -v

The --range and -v flags help narrow down the failure.

New output, seems like I was missing another dep:

ricardo@home:~/src/oil$ fc
./test/spec.sh alias --range 9 -v
alias.test.sh
case	line	dash	bash	mksh	zsh	osh	
  9	112	FAIL	FAIL	FAIL	FAIL	FAIL	Alias detection happens before expansion
case: 9
[dash stdout] Expected 'X\nstatus=127\n', got 'X\nstatus=1\n'
dash stdout:
X
status=1

dash stderr:
/bin/dash: 1: shopt: not found
Traceback (most recent call last):
  File "/usr/bin/e", line 9, in <module>
    import gconf
ImportError: No module named gconf


case: 9
[bash stdout] Expected 'X\nstatus=127\n', got 'X\nstatus=1\n'
bash stdout:
X
status=1

bash stderr:
Traceback (most recent call last):
  File "/usr/bin/e", line 9, in <module>
    import gconf
ImportError: No module named gconf


case: 9
[mksh stdout] Expected 'X\nstatus=127\n', got 'X\nstatus=1\n'
mksh stdout:
X
status=1

mksh stderr:
/bin/mksh: <stdin>[1]: shopt: not found
Traceback (most recent call last):
  File "/usr/bin/e", line 9, in <module>
    import gconf
ImportError: No module named gconf


case: 9
[zsh stdout] Expected 'X\nstatus=127\n', got 'X\nstatus=1\n'
zsh stdout:
X
status=1

zsh stderr:
zsh: command not found: shopt
Traceback (most recent call last):
  File "/usr/bin/e", line 9, in <module>
    import gconf
ImportError: No module named gconf


case: 9
[osh stdout] Expected 'X\nstatus=127\n', got ''
[osh status] Expected 0, got 1
osh stdout:

osh stderr:
usage error: Invalid option 'expand_aliases'
Unexpected error in execvpe('alias', ['alias', 'e=echo'], ...): [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/usr/bin/e", line 9, in <module>
    import gconf
ImportError: No module named gconf
Traceback (most recent call last):
  File "/usr/bin/e", line 9, in <module>
    import gconf
ImportError: No module named gconf
Traceback (most recent call last):
  File "/usr/bin/e", line 9, in <module>
    import gconf
ImportError: No module named gconf


0 passed, 0 ok, 0 known unimplemented, 0 known bugs, 5 failed, 10 skipped

FATAL: Got 5 failures (1 osh failures), but 10 are allowed

[andychu]

Hm interesting it looks like it's because you have a file called e in your $PATH.

OK let me think of how to adjust the test case not to be sensitive to that.

I guess I could just change it to echo_alias_ or something.

[andychu]

OK please pull master and try it again! Thanks for the report.

Awesome it works now, although I'm not seeing the file you are suggesting:

echo $PATH
/home/ricardo/.opam/system/bin:/home/ricardo/.local/share/umake/go/go-lang/bin:/home/ricardo/.local/share/umake/bin:/home/ricardo/bin:/home/ricardo/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/plan9port/bin

nvm found the offender
lrwxrwxrwx 1 root root 31 May 28 20:02 /usr/bin/e -> /usr/share/gedit/gmate/gmate.py

[andychu]

note: umask in octal in builtins.test.sh is flaky because sometimes rm outputs to stderr, depending on the initial test state.

This should be easy to fix.

[dash stderr] Expected u'', got "rm: cannot remove '/home/andy/git/oilshell/oil/_tmp/spec-tmp/builtins.test.sh/umask-one': No such file or directory\nrm: cannot remove '/home/andy/git/oilshell/oil/_tmp/spec-tmp/builtins.test.sh/umask-two': No such file or directory\n"

@granttrec I'm about to run your change and I noticed this. Thanks for the PR.

[andychu]

NOTE: I just had an instance of the redirect test being flaky under

devtools.sh/release.sh spec-all

Unfortunately this takes a long time. When I re-ran it with test/spec.sh redirect, it passed.

I just re-ran the whole thing and it succeeded the second time.

[andychu]

Also, background is flaky :-(

  File "/home/andy/git/oilshell/oil/osh/cmd_exec.py", line 1155, in _Execute
    status, check_errexit = self._Dispatch(node, fork_external)
  File "/home/andy/git/oilshell/oil/osh/cmd_exec.py", line 713, in _Dispatch
    status = self._RunSimpleCommand(argv, fork_external, span_id)
  File "/home/andy/git/oilshell/oil/osh/cmd_exec.py", line 577, in _RunSimpleCommand
    status = self._RunBuiltin(builtin_id, argv, span_id)
  File "/home/andy/git/oilshell/oil/osh/cmd_exec.py", line 276, in _RunBuiltin
    status = builtin.Echo(argv)
  File "/home/andy/git/oilshell/oil/osh/builtin.py", line 239, in Echo
    sys.stdout.flush()
IOError: [Errno 32] Broken pipe
JobState WhenDone 99

[andychu]

background issue should be fixed by: f68054e

Still need to fix the redirect issue.

[andychu]

The PWD test failed when doing the OSH 0.6.pre17 release? Couldn't repro it.

Not a flaky test! It caught a real bug. It was due to bin/osh being a shell wrapper and exporting $PWD, while the real osh is a symlink.

[andychu]

On the 0.6.0 release, case #1 of spec/redirect failed

1 <&

[osh stdout] Expected '[foo]\n', got '[]\n'

stdout:

[]

stderr:

  exec 5< $TMP/lessamp.txt
       ^~
[ stdin ]:2: 5: Bad file descriptor
  read line <&5
            ^~
[ stdin ]:3: 5: Bad file descriptor

It works when re-running that test serially.

[andychu]

Closing in favor of #995 and #330