bushmills / yoda

yoda is an acronym of Yet Onather Delirious Application, implementing a Forth compiler which translates Forth source code to bash functions.

yoda

yoda is an acronym of Yet Onather Delirious Application, and does naturally not refer to any greenish midget with pointy ears, albeit it had been said that reading yoda source aloud sounds like the very same midget talking.

yoda implements a compiler which converts a close resemblance of Forth source code to bash functions, an interpreter which runs them, and a base vocabulary of useful functions, strongly orientated towards what a common Forth interpreter offers. It can be used to augment bash, can serve as shell, or as Forthish coding environment.

yoda facilitates exploring it, by providing words for examining misc aspects, described on the Explore wiki page.

By now is yoda more standards compliant than I originally believed it to be feasable. Only few differences remain. Consequently has the pool of code intending to demonstrate the differences been ground down. Only few source code examples remain in Examples wiki page. By the same token has the extent of the file describing the deviations been reduced too: Differences wiki page.

My musings, ideas, plans and thoughts of possible changes can be found at Considerations wiki page, and the rest possibly among the remaining wiki pages.

yoda has been coded as bash script with only a small count of external dependencies

yoda's Issues

data items can't be ticked

variables, constants, arrays, but also words created with defining words containing does> can't be found when ticked.
Reason is the current representation of data items - there is no executable address associated with them, therefore tick can't return any. I notice that may be an undesirable restriction which may need a change or at least a workaround.

recurse didn't survive the change to late headers

most likely because the header it refers to, of the word which was in progress of compilation when recurse executed, didn't yet exist.
Will have to find a different way of self-referential, or let detokeniser deal with recursion.
Avoid using it for now. Only "fibo" demo and test recurse now.

Can't pipe input to yoda

yoda <<< "words
echo "words" | yoda
These will fail. While "words" will be initially executed, input doesn't catch the condition that it run out of it, and repeat querying.
It should "bye" instead.

yoda "words" will be accepted, and exit to calling shell after execution. This is the desired behaviour

recorded source location of colon words is that of semicolon

A consequence of late headers, which leaves creation of headers to semicolon, rather than to the defining word. Workaround shouldn't be too hard, but low priority now, as this isn't considered a huge problem.

The word `c@` does not fetch the character width

If the character size is equal to the cell size, then c@ shall be equivalent to @.
In Yoda it does not hold:

1 chars  1 cells  = .   \ -1
-1 pad !  pad c@  pad @  = .   \ 0

Why not just define c@ as:

: c@ ( c-addr -- char ) @ ;

For operations on octets, words like b@ and b! could be defined.

some characters refuse to be part of double quote multi word strings

a string like "${foo[1]} ${foo[2]}" will fail, apparently due to square brackets,

Error: "newprompt" is not found

The recent versions cannot start in my environment.

bash version: GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)

yoda version: 0.6.2, 2022-02-09, commit 901e2d8

The transcript is following:

./yoda
./yoda: line 2484: lib["$word"]: bad array subscript
needs:  newprompt
ERROR: not found: newprompt
at 1,18 of /work/forth.data/repo/github/yoda/defaults

According to git bisect, the first commit that shows an error is d41927b — it reports "ERROR: not found: files" but runs. The first commit that doesn't run REPL is c80d1f4.

Strange behavior of "previous"

A transcript is following:

./yoda
yoda shell
 ok
: foo previous ;
./yoda: eval: line 442: syntax error: unexpected end of file
 ok
foo
./yoda: line 825: yoda_1377: command not found

version: commit 3311098

Can't perform a real warm start which empties call stack but leaves dictionary intact

Therefore will errors and breaks (Ctrl-C) pile up new quit instances on top of those which got interrupted or error trapped.
Response of yoda to Ctrl-C can be set as preference in yoda.conv. This setting exists because of not having a clean warm start.
Current workaround: don't push Ctrl-C thousands of times.

Second best thing which I can currently do is a cold start instead, which will also reset dictionary to boot conditions (empty it, that is).
Not exactly the kind of response you wish to happen for a simple "word not found" typo.

Examples in README.md

It would be great to see examples of how to use Yoda, such as terminal transcripts, or minimal but functional source files.

source file and location information ("where") didn't survive change to late headers creation

also affects review. Source line is recorded incorrectly now for words defined in the yoda file. Probably minor effort to fix, just not highest priority atm.

words with lest or unless flow control may not be inlined

both lest and unless compile conditional exits, which break out of the function lest or unless were compiled into. When that code gets inlined into another word, it will exit from that word instead, rather than only from the word which was originally within lest or unless scope. For words having exitcompiled into (translated into a return from function), same applies.

workaround for now: don't mark words containing lest, unless or exit with inline.
fix thought of: lest, unless, exit mark the word as "may not inline", which is checked by inline, Upon attempt to tag such a word as inline, an error is thrown.

While wordlists/vocabularies allow multiple identically named but otherwise unrelated headers, flags don't

Therefore will the single instance header flags array only carry flags of the most recent version of identically named headers, overwriting those of prior headers in different wordlists.
Only the inline flag is used, therefore will this issue only manifest when a header in one word list is set to inline, while the identically named word in another word list is not set as such - resulting in word with earlier header not getting inlined any longer. Or vice versa.

While this can be viewed as a non-fatal potential nuisance, there exists (existed) also a condition with more grave effects: recent version of trashing headers would also trash the associated function if word was set to inline, assuming that no call to function body has been compiled. Due to this unreliability of inline header flag may that not be true
in all circumstances.
This condition has for the time being been dealt with by disabling trashing of function body in case header is tagged as inline.

Tag releases

Based on the contents of the "changes" directory, many versions of yoda have been released, yet there are no git tags in this repository. Could you please tag your releases?

Division in ANS Forth

The "Differences" web-page says at the moment:

specific words where deviating from ANS

/ mod /mod are truncating, not floored
I consider changing these to produce floored results

But actually it's not a deviation from ANS Forth (Forth-94), and from Forth-2012 too. According to these standards, whether the division is floored or symmetric is implementation-defined in the mentioned cases.

The only words for which a type of division rounding is specified are FM/MOD and SM/REM.

headers of words which failed compilation are kept

Those words are also executable, but due to failed compilation won't run successfully. I refrain from applying a quick fix such as a hide/reveal header flag, as there's another change pending which will address this issue too: Instead of creating headers, it is planned that : (and other defining words) pass on header creation to the detokeniser, passing name in the compile buffer, which gets processed only when
semicolon indicates completion. as detokeniser does merely translation of input to output, and returns output back to further processing only when it's done with the whole definition, it can then easily return nothing in case of failed compilation, including the header creating action.
Code for this is present, detokenizer has been prepared, but this is a change global to all words in yoda, and as such, some effort (and probably bug hunting) will be needed.
Most effective way to deal with the case of broken words but alive headers at this time is probably to trash them: trash name_of_defective_word

words mustn't be added by instant resolving during suspended compilation

example:
: foo [ epoch ] literal ;

assuming that epoch hasn't been compiled already prior to starting compilation of foo, instant resolving will start compilation of new word epoch while foo hasn't been completed.
As compilation isn't nestable, an error occurs.
Workaround:
satisfy epoch
: foo [ epoch ] literal ;

"postpone" does not work properly

postpone should append the compilation semantics of a word, but it appends the execution semantics.

yoda/yoda

Line 1504 in d9c3295

code 'code "${exists[0]}" "$call"'

Some testing:

: foo 123 . ;
: bar postpone foo  ;
see foo
see bar

Output:

found in forth:
yoda_1413 is a function
yoda_1413 ()
{
    ((s[++sp]=123));
    yoda_1323
}

found in forth:
yoda_1414 is a function
yoda_1414 ()
{
    yoda_1413
}

As it shows, bar just calls foo. But it should compile foo (since foo is an ordinary word):

{
    ((s[++sp]=1413));
    yoda_1342
}

See also the standard testcase for postpone.

words created by a defining word without code following does> will cause an error

example:
: foo create does> ;
foo bar
bar

very most likely the same issue which has been dealt with in : foo ; and empty flow control constructs like begin again, if else then, where I insert a ":" into the function as cheapest no operation replacement in bash.

It appears that cursor- and terminal control sequences don't work correctly with FreeBSD

Those are used for positioning the status line which is enabled by default, and for colour changes of input line and errors. Those terminal controls are taken from tput, and the problem may need more examination.
Disable input line colour changes and inclusion of statusline code by editing the file "defaults" in yoda directory: Remove "newprompt$" from the line which starts with "needs:"

"is" does not work properly

The compilation semantics for is are incorrect; they are default at the moment — since this word is defined as an ordinary word:

yoda/yoda

Lines 1243 to 1252 in d9c3295

	colon 'is'
	code 'word'
	code '[[ -z "$word" ]] && missingname'
	code 'exists "$word" "${order}" "$interpretonlywid" || notfound "$word"'
	code 'local context'
	code 'declare -n context="wordlist_${exists[2]}"'
	code 'tmp="$(type "${context[$word]}"|grep "${nameprefix}_.*m\[")"'
	code 'm["${tmp//[!0-9]/}"]="${s[sp--]}"'
	semicolon

And it doesn't parse anything in compilation state, but it should. See also the standard testcase for this word.