emersion / mrsh

A minimal POSIX shell

mrsh

A minimal POSIX shell.

• POSIX compliant, no less, no more
• Simple, readable code without magic
• Library to build more elaborate shells

This project is a work in progress.

Build

Both Meson and POSIX make are supported. To use Meson:

meson build/
ninja -C build/
build/mrsh

To use POSIX make:

./configure
make
./mrsh

Contributing

Either send GitHub pull requests or send patches on the mailing list.

License

MIT

mrsh's Issues

Variables that aren't valid names don't work in ${variable}

$ echo ${67432}
mrsh:1:8: syntax error: expected a parameter

Arithmetic operations

Job control

`false` sets last_status to 0

false
echo $?

It seems that the false command is creating five distinct task lists, and the genuine status gets lost in the muck.

mrsh status

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html

• Parser
  • Basic line parser
  • Comments
  • Quoting, escaping
  • Aliases
  • Parameters, variables
  • Word expansions
    • Parameter expansion
      • Variables
      • Expressions
    • Command substitution
      • `…`
      • $(…)
    • Arithmetic expansion
  • Redirections
    • Redirecting IO
    • Here-document
  • Pipelines
  • Async lists
  • Compound commands
    • {…}
    • (…)
  • Control structures
    • if, elif, else, fi
    • while, for, until, do, done
    • case, in, esac
  • Function definition
• Shell
  • Simple commands
  • Redirections
    • Redirecting IO
    • Here-document
  • Pipelines
  • and-or lists
  • Async lists
  • Compound commands
    • {…}
    • (…)
  • Parameters, variables
    • Shell variables
    • Positional parameters, special parameters
  • Word expansions
    • Tilde expansion
    • Parameter expansion
      • Variables
      • Expressions
    • Command substitution
    • Arithmetic expansion
    • Field splitting
    • Pathname expansion
    • Quote removal
  • Control structures
    • Conditions
    • Loops
    • Case
  • Aliases
  • Function definition
  • CLI flags
  • Job control
  • Exit status and errors
  • Signals and error handling
  • Pattern matching
• POSIX build system
• Special builtins
  • break
  • : (colon)
  • continue
  • . (dot)
  • eval
  • exec
  • exit
  • export
  • readonly
  • return
  • set
  • shift
  • times
  • trap
  • unset
• Regular builtins
  • type
  • ulimit
  • alias
  • bg
  • cd
  • command
  • false
  • fg
  • getopts
  • hash
  • jobs
  • pwd
  • read
  • true
  • umask
  • unalias
  • wait

Variable expansion

• Parser
• Shell

Improve parser errors

Right now it's just a bunch of assertions.

We need to track line numbers too.

Terminating a command with `&` peeks next char

echo hey &

Should start the command immediately. Instead it waits for the next one. It executes both after the next newline.

Non-interactive command substitution repeats subsequent commands twice

echo $(echo 1)
echo 2

Output:

1
2
2

A few memory leaks

#!/bin/sh
n=asdf
echo start
while [ "$n" != "fdsa" ]
do
    echo "n: $n"
    n="fdsa"
    echo "n: $n"
done

==11074== HEAP SUMMARY:
==11074==     in use at exit: 893 bytes in 23 blocks
==11074==   total heap usage: 565 allocs, 542 frees, 88,521 bytes allocated
==11074== 
==11074== 223 (88 direct, 135 indirect) bytes in 1 blocks are definitely lost in loss record 18 of 20
==11074==    at 0x483BCE2: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11074==    by 0x4851702: mrsh_simple_command_create (ast.c:283)
==11074==    by 0x4852C47: mrsh_command_copy (ast.c:723)
==11074==    by 0x485CBF4: copy_simple_command (tasks.c:10)
==11074==    by 0x485CC30: task_for_simple_command (tasks.c:19)
==11074==    by 0x485D092: task_for_command (tasks.c:93)
==11074==    by 0x485D216: task_for_pipeline (tasks.c:119)
==11074==    by 0x485CF7B: task_for_node (tasks.c:135)
==11074==    by 0x485CEDD: task_for_command_list_array (tasks.c:64)
==11074==    by 0x485C925: mrsh_run_program (shell.c:34)
==11074==    by 0x109824: main (main.c:126)
==11074== 
==11074== 303 (88 direct, 215 indirect) bytes in 1 blocks are definitely lost in loss record 19 of 20
==11074==    at 0x483BCE2: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11074==    by 0x4851702: mrsh_simple_command_create (ast.c:283)
==11074==    by 0x4852C47: mrsh_command_copy (ast.c:723)
==11074==    by 0x485CBF4: copy_simple_command (tasks.c:10)
==11074==    by 0x485CC30: task_for_simple_command (tasks.c:19)
==11074==    by 0x485D092: task_for_command (tasks.c:93)
==11074==    by 0x485D216: task_for_pipeline (tasks.c:119)
==11074==    by 0x485CF7B: task_for_node (tasks.c:135)
==11074==    by 0x485CEDD: task_for_command_list_array (tasks.c:64)
==11074==    by 0x485E826: task_loop_clause_create (loop_clause.c:67)
==11074==    by 0x485D1B4: task_for_loop_clause (tasks.c:85)
==11074==    by 0x485D124: task_for_command (tasks.c:105)
==11074== 
==11074== 303 (88 direct, 215 indirect) bytes in 1 blocks are definitely lost in loss record 20 of 20
==11074==    at 0x483BCE2: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11074==    by 0x4851702: mrsh_simple_command_create (ast.c:283)
==11074==    by 0x4852C47: mrsh_command_copy (ast.c:723)
==11074==    by 0x485CBF4: copy_simple_command (tasks.c:10)
==11074==    by 0x485CC30: task_for_simple_command (tasks.c:19)
==11074==    by 0x485D092: task_for_command (tasks.c:93)
==11074==    by 0x485D216: task_for_pipeline (tasks.c:119)
==11074==    by 0x485CF7B: task_for_node (tasks.c:135)
==11074==    by 0x485CEDD: task_for_command_list_array (tasks.c:64)
==11074==    by 0x485E972: task_loop_clause_poll (loop_clause.c:50)
==11074==    by 0x485EFE1: task_poll (task.c:30)
==11074==    by 0x485EAE0: task_pipeline_poll (pipeline.c:69)
==11074== 
==11074== LEAK SUMMARY:
==11074==    definitely lost: 264 bytes in 3 blocks
==11074==    indirectly lost: 565 bytes in 19 blocks
==11074==      possibly lost: 0 bytes in 0 blocks
==11074==    still reachable: 64 bytes in 1 blocks
==11074==         suppressed: 0 bytes in 0 blocks

What does this do?

More info in README?

Don't perform pathname expansion on quoted strings

$ mkdir "*"
$ touch "*/thing"
$ ls "*/"*
'*/thing'

Segfault on assigning variable to empty value

• open a shell, enter CC= (or any other variable)
• expected behaviour: the shell sets CC to the empty string (or unsets it)
• actual behaviour: segfault (assign->value is NULL at ast.c:665)

Line break missing

Hi,

From your favorite shell:

build/mrsh

Then, from mrsh:

build/mrsh

Then, exit the last one. You get something like $ $ printed, because a line break is missing.

Spaces in arithmetic expressions

echo $((1 + 2))

Exit in loops body doesn't exit immediately

while true
do
    exit
    echo bad
done

Should also try exit in conditions.

Don't allow unquoted chars

The application shall quote the following characters if they are to represent themselves:

|  &  ;  <  >  (  )  $  `  \  "  '  <space>  <tab>  <newline>

Associativity in arithmetic expressions

Some operators have the wrong associativity. See

• https://en.cppreference.com/w/c/language/operator_precedence
• https://eli.thegreenplace.net/2009/03/14/some-problems-of-recursive-descent-parsers
• https://github.com/mvdan/sh/blob/master/syntax/parser.go#L1061

Aliases ending in a blank should make substitution continue

$ alias hey='ls '
$ alias there='-a'
$ hey there

Should yield ls -a.

Push argc/argv on function call

And pop on function return. We probably need a stack.

Arithmetic expressions with shifts are not parsed

>> and << stop the word parser.

Alias substitution

Only happens when parsing command names. Happens before grammar rules are applied, so can expand to reserved words and… anything really.

$ alias hey='  '
$ alias there='if true; '     
$ hey there then echo a; fi
a
$ alias hi='a=b env |'
$ hi grep a=
a=b
$ alias if=lol
$ if
> dash: 3: Syntax error: end of file unexpected
$ a=b if
dash: 3: lol: not found

Variables unset in command substitution subshells

echo `echo $PPID`

Shell exit status

The following exit values shall be returned:

0 The script to be executed consisted solely of zero or more blank lines or comments, or both.
1‐125 A non-interactive shell detected an error other than command_file not found, including but not limited to syntax, redirection, or variable assignment errors.
127 A specified command_file could not be found by a non-interactive shell.

Otherwise, the shell shall return the exit status of the last command it invoked or attempted to invoke (see also the exit utility in Section 2.14, Special Built-In Utilities).

Reimplement wordexp

• https://git.musl-libc.org/cgit/musl/tree/src/misc/wordexp.c
• https://github.com/Apple-FOSS-Mirror/Libc/blob/2ca2ae74647714acfc18674c3114b1a5d3325d7d/gen/wordexp.c#L192

Add mrsh_parser_with_buffer

It would read from a buffer shared with the caller. This would allow us to have a nicer main.c.

Function test fails on FreeBSD

See https://builds.sr.ht/~emersion/job/9653

Handle `$\n` more gracefully

The spec says it's undefined behaviour. Currently we interpret this as a variable named \n. We should probably handle this as the text $.

Set $LINENO

$ echo $LINENO; \
> echo $LINENO 
1
2

Should annotate each simple command with a line number. Should set $LINENO before executing those.

Rename mrsh_token to mrsh_word

"Token" is too broad.

Parsing error in function redirection

#!/bin/sh
file=example_1

example() {
	echo hello world
} > $file

example
file=example_2
example

Show AST

~/s/m/build > ./mrsh -n test.sh
program
program
└─command_list ─ pipeline
  └─simple_command
    └─assignment
      ├─name file
      └─value ─ word_string [2:6 → 2:15] example_1
program
program
├─command_list ─ pipeline
│ └─function_definition example ─ brace_group
│   └─command_list ─ pipeline
│     └─simple_command
│       ├─name ─ word_string [5:2 → 5:6] echo
│       ├─argument 1 ─ word_string [5:7 → 5:12] hello
│       └─argument 2 ─ word_string [5:13 → 5:18] world
└─command_list ─ pipeline
  └─simple_command
    └─io_redirect
      ├─io_number -1
      ├─op >
      └─name ─ word_parameter
        └─name file
program
program
└─command_list ─ pipeline
  └─simple_command
    └─name ─ word_string [8:1 → 8:8] example
program
└─command_list ─ pipeline
  └─simple_command
    └─assignment
      ├─name file
      └─value ─ word_string [9:6 → 9:15] example_2
program
└─command_list ─ pipeline
  └─simple_command
    └─name ─ word_string [10:1 → 10:8] example

Expected: the redirection should be a part of the function node. It will need to be stored with the function and evaluated each time the function is called.

Assertion failed when syntax error in back-quotes

$ echo `if`

mrsh: ../shell/task_command.c:93: void get_args(struct mrsh_array *, struct mrsh_simple_command *, struct context *): Assertion `fields.len > 0' failed.

IO redirections missing from function definitions

May be missing in other places too.

Subshells do not exit with correct status code

( exit 1; ) && echo "shouldn't happen"

PS1 contains multiple tokens

We only parse the first one.

$ PS1='hey there'
hey

Tilde expansion in assignments

Should expand after : too, not just at the beginning of the string.

Set $PPID

Set by the shell to the decimal value of its parent process ID during initialization of the shell. In a subshell (see Shell Execution Environment), PPID shall be set to the same value as that of the parent of the current shell. For example, echo $ PPID and (echo $ PPID) would produce the same value.

Remove running_processes global state

Move this into mrsh_state. Might want to use linked lists instead of arrays.

Make the parser nicer

Probably just ignore what the standard says about lexing.

The following script loops forever

#!/bin/sh
n=asdf
echo start
while [ "$n" != "fdsa" ]
do
    echo "n: $n"
    n="fdsa"
    echo "n: $n"
done

Aliases mess up line/column numbering

Path expansion

Continuations lines cannot be inserted inside operators/keywords

false &\
& echo a
# same as:
false && echo a

Split parser into multiple files

It's getting pretty big.

Parse quotes and candidates for expansion/substitution as a tree

That means, don't unquote when parsing. That probably means: keep a list of {unquoted, quoted} fragments when parsing.

Parse ${#parameter}

Should evaluate to the length of the variable parameter.

Tool which parses a script and writes the script back out

Should test that the AST contains enough information to go from script -> AST -> script losslessly. Checksums should match before and after.

Move away from FILE

Don't use libc's FILE pointers. We already have a buffering system in place, and we don't need to use fmemopen since we can create a parser from a buffer.

This would allow us to undo 51c7fe8 and remove state.input.

Here-documents

Should parse things like:

{ echo a; cat <<EOF1; cat <<-"EOF2"; ls; } & \
echo hey
This is 1.
Double-quotes are ignored: "$HOME
Substitutions are performed: $(ls)
EOF1
	This is 2.
		Substitutions are not performed: $(ls)
	EOF2

We probably want to add a list of here-documents in the parser state, and read those after a complete command has been parsed.

Add mrsh_word_append_to

mrsh_word_append_to(word, buffer) would write to a buffer. This would allow to optimize when calling mrsh_word_str in a loop.