sagebind / riptide Goto Github PK
View Code? Open in Web Editor NEWThe Riptide Programming Language: Shell scripting redesigned.
Home Page: https://riptide.sh
License: MIT License
The Riptide Programming Language: Shell scripting redesigned.
Home Page: https://riptide.sh
License: MIT License
"Shared" values such as tables and strings are currently using naive reference counting, but cycles for circular tables are not checked at all and are not cleaned up until the end of the program. This might be fine for short scripts, but could cause long-running interactive shells to have continually growing memory usage if cycles are created.
Since preventing cycles in the first place is not an option (you're always allowed to assign a table as a field inside another table), we should consider using a more sophisticated garbage collection scheme to bust reference cycles once no longer in use. There are several options for doing this today:
Strings can continue to use reference counting, or even something a little different (string interning) since they have different usage patterns and can't have cycles.
The built-in logger currently does not play nice with standard error. Since we make standard I/O handles non-blocking while the Riptide runtime is in control, the logger will rightfully get an EAGAIN
or equivalent error from time to time when trying to write a log and the buffer is full. We should update the logger to either:
The latter seems like the more correct solution, but might be trickier, since loggers are global resources.
Currently we are using termion, which has an OK API, but like almost every terminal-related library I can find, is inherently synchronous. This won't work for us, because interactive mode will be driven using a single-threaded event loop, and input events need to be read asynchronously.
Here's the plan:
The workflow ci.yml is referencing action actions/checkout using references v1. However this reference is missing the commit a6747255bd19d7a757dbdda8c654a9f84db19839 which may contain fix to the some vulnerability.
The vulnerability fix that is missing by actions version could be related to:
(1) CVE fix
(2) upgrade of vulnerable dependency
(3) fix to secret leak and others.
Please consider to update the reference to the action.
It would be a useful distinction to be able to page through just the command history for a single session, or for history of all sessions.
To do this we have to figure out what a "session" is:
To avoid duplicate storage, we could create a new table that is basically session_history
and have command_history
entries hace a foreign key pointing to the originating session. We could move pid
from the command entries to the session entry in this scenario assuming a session can't span multiple PIDs.
Are forks the same session, or a new one?
As someone who used to be heavily involved in the development of projects like Oh My Fish! for many years, I've learned a lot about shells, package distribution, plugins, and themes, and various pitfalls that should be avoided. One such area that we must be careful with are ways of customizing and augmenting the terminal editor (prompt) itself.
When a "plugin" is simply a bundle of scripts that can hook into just about anything it means that plugin authors can do whatever they want, which is nice from one perspective, but it also causes a slew of other problems for the shell itself:
In attempt to avoid some of these, here's what I'd like Riptide's extensibility to look like:
The rustyline library implements history, word completion, multiple lines,
etc. I'm not sure if it would be a good idea to use it in a project that wants
to become it's own shell, but it would be a significant step up in features for
riptide currently.
I'm willing to do some of the legwork here, but I want to know if there's a
reason not to before I do.
For example, this should be allowed:
echo (require lang)->VERSION
but currently only string literals are allowed in a variable path.
At some point when Riptide becomes usable we will need a snazzy home page that explains what Riptide is and links to the guide and documentation.
While the syntax parser is practically complete (and well designed too), the interpreter is functional but still needs quite some work. Currently pipelines with more than one step (as in, a regular statement) are ignored by the interpreter with a TODO
. There are several reasons for this, but the main reason is that there has been no clear implementation path on how to run pipeline steps in parallel.
There are several possible ways of parallelizing pipelines:
In this approach, you set up N-1 anonymous pipes, where N is the number of steps, fork N-1 times, and set the stdin/stdout of the child processes to use the anonymous pipes in order to form a chain. Then you poll the right-most step, or execute it directly if it a shell function. This is the traditional approach taken by many shells, including popular ones like Bash and Z shell.
The advantage of this approach is that it unifies the pipeline implementation; you always pipe multiple processes together, whether they are external commands, or forks of the shell. The biggest downside is that the subshells are disconnected from the scope they came from and cannot (easily) communicate with the parent. This can cause lots of confusion when learning how to write scripts in these languages, and so I decided that this approach was unacceptable for Riptide. Take the following example code:
def count 0
ls | grep foo | each {
set count (+ $count 1)
} > foo-files.txt
println "Found $count foo files"
While you might expect the above script to print out Found 10 foo files
if there are 10 files matching in the current directly, instead you will always get Found 0 foo files
, because the each
block has to operate on a clone of the count
variable. The original count
never gets updated.
Another approach is to use background threads or a thread pool to run multiple steps in a pipeline in parallel. "Script" steps are run in threads, while external commands are run in real processes. We then use abstractions inside these threads to make them behave as if they were normal processes. The current variable scope is then shared between these threads behind a mutex, so that they can all mutate their scope normally. Interestingly, this seems to be the approach that Fish Shell takes.
This lets you pipeline multiple script blocks together and be able to have the expected "normal" scoping rules for each, but has several disadvantages:
exec
. You have to jump through a lot of hoops to avoid these gotchas.This is the approach I propose we use in Riptide (see #11). We turn every potentially parallel bit of code into an asynchronous task, and then execute all tasks on the main thread using a single-threaded executor. This has many benefits:
This will require some refactoring, since the interpreter must essentially be asynchronous and be able to "yield" in the middle of a script and be resumed later. Typically this would be incredibly difficult to do using a tree-walk interpreter, but Rust's async/await makes this almost trivial by generating the complex state machine for us.
We could also accomplish the same thing by rewriting the interpreter into a JIT VM, but that would be quite a bit more work and would increase the complexity of the interpreter by quite a lot. I'd rather take tree walking as far as it can possibly go in order to keep the implementation simple, and only reconsider if it is an actual bottleneck.
For Riptide to be useful at all, a minimal list of builtin functions and constructs must be provided in order to be able to program properly. Below is a list of functions that should be implemented before an alpha release:
=
: Test equivalence.|
: Create a pipeline.and
: Logical and.apply
: Function call argument unpacking.begin
: Execute expressions in sequence.builtin
: Explicitly call a builtin.capture
: Capture stdout and return it.cd
: Change directory.command
: Explicitly call a command.cond
: Multiple conditional branching.def
: Define a global function or variable (binding).let
: Define a local, lexically scoped binding.env
: Get, set, or list environment variables.exec
: Replace current process.exit
: Exit program.foreach
: Iterate over a list.help
: Some sort of help system.if
: Conditional branching.list
: Create a list.not
: Negate a boolean.nth
: Return nth item in list.or
: Logical or.print
: Write to output.pwd
: Print current directory.read
: Read from input.source
: Evaluate a script file.(update this list as needed)
The interpreter should be refactored to be fully asynchronous, using Rust futures and executors under the hood. We gain several huge wins by doing this:
Since Riptide is still young, it is acceptable to switch to using nightly Rust for the interpreter to take advantage of any features we need to implement the async interpreter (though most notably async/await).
It would be very handy if regex was built into the language syntax itself as an embedded language mode. This would give you syntax errors to catch early if you have a mistake in your regex! (Of course also allow creating a regex from a string.)
I imagine the syntax looking something like this:
# If matched, $results is a list of capture groups. If not, then nil
$results = "my string" =~ re#\w+#
The tricky part is that forward slashes are often used as delimiters, but those mean directory separators far more often in a shell.
The workflow publish-docs.yml is referencing action actions/checkout using references v1. However this reference is missing the commit a6747255bd19d7a757dbdda8c654a9f84db19839 which may contain fix to the some vulnerability.
The vulnerability fix that is missing by actions version could be related to:
(1) CVE fix
(2) upgrade of vulnerable dependency
(3) fix to secret leak and others.
Please consider to update the reference to the action.
Make stdin, stdout, and stderr into normal context variables and make a "stream handle" a built-in type. Then we can do something like:
let @stdout = (open -w "message.txt") {
println "Hello world!"
fclose @stdout
}
All sorts of fun and useful programming abilities open up that many shells can't handle because you can't hold a file descriptor in a variable. Of course, it can also mean that users can leak file handles...
We gain very little by making everything into a builtin and we lose some degree of convenience. The following things should be made first-class language constructs:
require
: Define some sort of import
syntaxif
cond
while
def
: This currently uses hacks to be implemented as a builtin.set
: This doesn't exist yet because it can't be implemented as a builtin.catch
throw
I don't want to compromise Riptide's asynchronous design by introducing dirty hacks in order to support Windows, but it would be cool for Riptide to work as a native Windows shell, as there's a poor few number of shells for it.
The old Win32 APIs for console applications does not allow asynchronous operation, but the new ConPTY API in Windows 10 does. We could build on top of that in order to have first-class WIndows support.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.