awto / effectfuljs

JavaScript embedded effects compiler

JavaScript embedded effects compiler

This is a tool to build JavaScript to JavaScript transpilers (babel plugins) along with a few predefined ones. It extends JavaScript language with various computation effects by means of runtime libraries but without any syntax extension.

There are such libraries for:

• EcmaScript effects extensions (async functions, generators and async generators)
  • Why not native or other transpilers
    • Abstract API - @effectful/es
    • Persistent state - @effectful/es-persist
    • Concrete API - @effectful/es-inline, the best performance
    • Implicit parallelism (experimental)
    • Deriving program's static graphs for analysis and conversion to other languages - WIP
• Multi-prompt delimited continuations - @effectful/cc
• Debugger API - @effectful/debugger and VSCode plugin effectful/debugger
• Workflow-as-Code on Kafka - kafka-workflow

Not yet implemented:

• algebraic effects
• probabilistic programming
• parallel and distributed programming
• persistent continuations
• adaptive computations
• logical programming - Old version
• reactive programming (with RxJS) - Old version

They are typically small libraries, some of them are just tiny wrappers around well-known interfaces, such as Promises or Observables.

The compiler converts ES2018 to ES2018 and doesn't need any syntax extensions. So it may be applied to results of other compilers targeting JavaScript such as CoffeeScript, TypeScript etc.

Usage

EffectfulJS is a tool to build transpilers. There are many packaging options and usages, but typically the follow next pattern. Transformations usually contain a babel plugin (and/or macro) along with some runtime. They can be in a single package, with Effectful to be a peer dependency (along with a dev dependency). This way the transpiler can be installed with:

$ npm install --save <lib name>
$ npm install --save-dev @effectful/core

If the babel plugin is in transform.js of <lib name> package, .babelrc can be:

{
   "plugins": [<lib name>/transform]
}

The package can also contain macro.js file for zero-configuration using babel-plugin-macros. This works also for Create Reat App) where .babelrc is disabled.

Check the libraries documentation for exact details:

• @effectful/js compiles to the most abstract code, single or double level depending on parameters
• @effectful/es for two-levels syntax either concrete ECMAScript compatible effects or abstract for async, generator and async generators semantics overloading.
• @effectful/es-inline A shortcut for @effectful/es with concrete ECMAScript effects implementations.

All the plugins and macros are just shortcuts to @effectful/core setting needed lower level options described in config.js thus generated code uses different runtime interfaces.

Examples

These examples are not the full list of features. The project is abstract, with a lot of different applications.

Persistent state

Save all your application state in DB or file. The state includes the point where it executes now along with all variables values implementing any serialization interface.

Restore the saved state after, maybe in a different process or even on a different hardware.

For the application, the state transfer will be transparent.

This feature greatly simplifies defining long-running workflows. If the program requires some not immediate action from a human, e.g. email-confirmation, or it should await some exact date. No needs to bother implementing all these details, just write a very simple script describing the business logic and nothing else.

Here's some shop business logic example.

function order(customer, item) {
  begin();
  customer.balance -= item.price;
  item.vendor.balance += item.price;
  scheduleShipment(customer, item);
  commit();
}

The scheduleShipment function may require some other people involved. They can reject the transaction and so the balances values must be reset to original levels, even if a few days already passed. We don't care about implementation details. Such as where and how to store the state, how to reset it, etc. Everything is done by some underlying generic framework. The framework is not a subject of EffectfulJS though.

There are a lot of other applications for a persistent state. It can be used also for debugging purposes. This makes it very simple to implement hot reloading and time traveling, as this just saving and restoring the application state at some particular execution points.

More details are in @effectful/es project.

Delimited continuations

This is the most generic computational effect. Any other can be implemented using it. It looks very similar to Coroutines (async functions and generators). But unlike Coroutines, it allows resuming computation from the same point more than once.

Unlimited continuation like callcc is a special case. They allow capturing all and only all execution context. While with delimited continuations it is possible to specify only some region. For example, for the persistent state, this allows saving not the whole application state, which may be big, but only some interesting part.

Check @effectful/cc for more details.

Implicit parallelism

Async functions considerably simplify asynchronous source code. But it serializes operation execution. Next operation executes only after the current is finished. If we want to run the operations in parallel we can use Promise.all.

Say, we have two async operations (probably some remote server requests) - A and B:

async function a() {
  const a = await A;
  const b = await B;
}

If these operations are independent they can be executed in parallel:

async function a() {
  const [a, b] = await Promise.all([A, B]);
}

This doesn't look more complex, but what if we make B dependent on A and add another operation C:

async function parDemo() {
  const a = await A;
  const b = await B(a);
  const c = await C;
}

Now it is impossible to start B before A is finished, but we can start C in parallel with A and B:

async function parDemo() {
  const [a, c] = await Promise.all([A, C]);
  const b = await B(a);
}

This version is not effective, operation C may take much longer than A and this will prevent B from starting execution. The correct version is:

async function parDemo() {
  let a;
  const [b, c] = await Promise.all([
    (async () => {
      a = await A;
      b = await B(a);
    })(),
    C
  ]);
}

This is obviously significantly more complex, error-prone, harder to read and maintain. It will become even much worse with some next small changes. They may require almost full function rewrite.

Fortunately, EffectfulJS can do the transformation automatically, just specify which block you want to parallelize. The transpiler will automatically compute variables dependencies and structure the block parts in Promise.all accordingly.

Here is how the last example looks with EffectfulJS:

async function parDemo() {
  "par";
  const a = await A;
  const b = await B(a);
  const c = await C;
}

Switching between parallel and sequential code is just a matter of adding "seq"/"par" directives.

This works for loops too. With the help of a persistent state some jobs may be delegated to WebWorker or some cloud server, making simple but effective distributed programs.

Reactive programming (WIP)

Reactive programs operate with values which change with time. While in the source code it looks like a single value variable:

function drawBox(root) {
  const down = event(root, "mousedown");
  if (down && down.button === 0) {
    const move = event(root, "mousemove");
    const up = event(root, "up");
    const cur = move || up;
    const box = (
      <div className={up ? "box" : "drawing"}>
        style=
        {{
          left: Math.min(down.x, cur.x) + pageXOffset,
          top: Math.min(down.y, cur.y) + pageYOffset,
          width: Math.abs(down.x - cur.x),
          height: Math.abs(down.y - cur.y)
        }}
        >
      </div>
    );
    const del = event(box, "contextMenu");
    if (!del) root.appendChild(box);
    if (!move) return;
  }
}

Here all the variables may change when some mouse event is received, while the program looks like it is executed only once. The framework is responsible for recalculating dependant parts of the program. Such programs are much easier to write, test, debug and maintain.

Logical programming (WIP)

This is a library for adding logical programming languages features to JavaScript. Most Prolog books start from classical bi-directional list append function. Depending on input arguments it may either append two lists or split some list in two. Here is an almost literal translation of its implementation in JavaScript:

function append(a, b, r) {
  const [h, t, rt] = newRefs();
  unify(a, cons(h, t));
  unify(r, cons(h, rt));
  append(t, b, rt);
  M.or();
  unify(a, nil());
  unify(r, b);
}

For front-end, this, for example, can be useful to specify some control's values constraints. In the next example, there are three cyclically dependent fields, namely payment amount without discount, with discount and the discount value. The end-user may enter two of them while the third will be calculated automatically.

function PaymentForm() {
  const [cost, discount, finalCost] = newRefs();
  constraint(finalCost === (cost * discount) / 100);
  return (
    <form action="/pay">
      <input type="text" name="cost" value={cost} />
      <input type="text" name="discount" value={discount} />
      <input type="text" name="finalCost" value={finalCost} />
      <input type="submit" value="Pay" />
    </form>
  );
}

The same can be done with the new React hooks, which is an implementation of computational effect relying on runtime only. Transpiler like EffectfulJS doesn't have runtime-only limitations, namely, React Hooks Rules. They significantly reduce the number of useful use cases. Some complex program logic may be simplified with loops, conditions and other common JavaScript constructs.

How it works

The compiler stratifies input JavaScript code into two levels. The first level (pure) contains JS constructs (expressions, statements) kept as is in generated code. The second level (effectful) contains effectful constructs. The compiler converts them into abstract function calls.

I'm abusing term pure here. The statements or expressions on this level can have any side effect already implemented in JavaScript. Namely IO, references, exceptions, loops, breaks, returns, etc.

The levels may be either explicit or implicit in syntax (two-level or single-level syntax resp.).

Async functions or generators syntax can be used for marking expressions of explicit effectful level.

async function a() {
  console.log("x:", await getX());
}

Unlike standard ECMAScript async function this will be converted to abstract API calls. The API is

The overloading may be either compile time - implementing some syntax transforms, or runtime - providing libraries with abstract interfaces implementations.

This is an example of one of a few possible outputs:

function a() {
  return M.chain(getX(), _1);
  function _1(a) {
    console.log("x:", a);
  }
}

There chain, raise methods are provided by some runtime library. For example, its concrete implementation for Promises calls then function for chain and rejects the promise in M.raise.

The names and interfaces of the functions in the generated code are configurable.

There is a dozen of such functions required to be implemented in concrete effects implementation library but, usually, most of them are trivial.

The interface is based on Monads. However understanding Monads is required only to implement some new effects library. No needs to know anything about them to use the libraries.

With the implicit single-level mode the same code may be even more succinct:

function() {
  console.log("x:",getX());
}

There are more examples of input/output in the test folder.

It is arguable if explicit or implicit levels separation is better. This likely depends on what kind of effects is used. The succinct and more readable code is good, but if effects are heavy making them explicit may be better. So EffectfulJS compiler supports both options.

LICENSE

Distributed under the terms of The MIT License (MIT).

effectfuljs's Issues

Reflect wrappers

Monkey patching of Reflect API to support time traveling (record reverse actions)

Inline more runtime operations

The generated code has almost no closure captured values, so it can be changed as a string in runtime and evaled back. This way it can inline some operations depending on the current runtime parameters.

console.time

console.time should redirect its output into DAP

Data breakpoint

DAP data breakpoints implementation.

PnP modules loading

Instrument modules loading via PnP

Stepping into setters/getters/Coercions/Proxies etc

Now the debugger can only step into another explicit function calls. While there are many situations where an implicit function is called, it will be executed but the debugger cannot step into it or stop on a breakpoint inside it.

To implement this all the operators which can involve the implicit calls must be converted into runtime functions calls, and in the runtime, the sequence of implicit calls should be done according to the ECMA standard.

Jest zero config

Jest prevents typical node modules hacks for running instrumentations on import. This way the traspilation should be run through jest transform and so requires configuration. Instead it could apply similar to create-react-scripts zero-config approach but adding the debugger's transform.

User space debugger

Since effectful.js can capture a delimited execution context along with its variables, this unlocks a lot of possibilities for diagnostics.

For example, we can trace the variable's data with ES Proxies and store current continuation along the trace. This continuation can be restored after and thus we get time traveling. Hot reloading also can restore closure captured variables. Remote requests memoized and during replay older value is returned. And so on.

dynamic imports

Dynamic imports aren't supported yet, but this is not something which can be added using babel's plugin and mast be implemented in the framework.

`with` statement implementation

The local variables in the generated code are accessed by index, this won't work for with statement where the scope is dynamic. To solve this it needs to use runtime functions for searching variables by name from the current scope if it is inside with statement.

Lazy instrumentation

During debugging it is highly unlikely all the functions from the application are called. To reduce compilation time the babel plugin on the first stage could just apply for closure conversion pass, and all the rest is done on the first function's call lazily.

This way if not in debugging mode it can run original functions without instrumentation, almost without any performance loss.

jsdom integration

The biggest part of debugging front-end application is debugging event handlers. Browsers expect the handlers to be executed synchronously but in the breakpoint, the debugger releases the thread, thus if there is something like event.stopPropagation after the breakpoint the propagation won't be stopped.

To avoid implementing all the standard propagations algorithms we can utilize jsdom where these are already implemented. This won't solve default actions problem still though (due to security restriction browsers don't give access to this functionality to scripts).

However, this would resolve target.dispatchEvent calls.

Proper hot reloading

The partly changed functions can be reloaded with saving their state but the execution position and variable's positions are quite random (position based rather than better name based). So for currently executing functions, this works well only if something very small, e.g. constant is changed only.

To resolve this it could hash the function's effect frames (e.g. using SHA-1) rather than just the frame's number, so than reloading it is easier to identify the exact position, and also the number of frames will be reduced.

More examples

Conditional debugger breakpoint?

Is it possible to do something like this in JavaScript:

for (var i = 0, l = tableNode.table.body.length; i < l; i++) {
+    if (i === 38) debugger;
    // ...
}

Or something like this in general in VS Code:

@effectful/serialization does not work in browser because `global` is missing

rather than using global directly, it should either use globalThis (supported in node and browser) or typeof window === "undefined" ? global : window

Types

Is it possible to support Flow/TypeScript?

Completion, expressions result for side effects free exceptions

Some expressions can be evaluated if they don't change the program state. This way, for example, in browser expression's results is immediately seen while we type it.

This debugger can do even better. It can run expression even if it has some side effects but reverting them after it has the result available.

changing parent scope in `eval`

Expressions like eval("var m = 10") should add variable` into its parent scope. This doesn't work now (if the variable isn't already defined in that scope), because the variables are accessed by index not by name.

To make this work each frame should also store reference of a global object, and each global variable access is converted to member access expression in that object. If some function uses eval it should also Object.create(parentGlobal). This way the eval will add the variable into the parent's global object and it can be accessed.

Blocking breakpoints

In the current debugger implementation, the main thread is released on breakpoints. This way we can make other requests (such as some expressions evaluation, reloading sources etc).

However, it is a problem for not-transpiled or native code which calls the functions synchronously and requires everything is done before exiting the function (for example event.preventDefault called in the event handler).

We still can block the thread in these situations (in browser's main thread deprecated but still working sync XMLHttpRequest, and in node or WebWorker using Atomics.wait, or even resorting to the native debug API sometimes).

Function breakpoints

Implement function's breakpoints according to the DAP protocol

Adaptive (Incremental) programming effects library

One of many possible effects EffectfulJS may support is an Adaptive (Incremental) programming. Namely re-executing only parts of the code when some dependency changes. Like Mobx @computed property but not re-executing the whole function, only a part (delimited continuation) to reflect the changes in variables. These may be loops, other functions calls etc - some heavy computations.

For example, if there is a huge data grid and we want to render only a visible part of it (with possibly applied filtering, sorting etc). We can write simple for-of loop rendering the grid between starting and finishing position, and if the variables change (on scrolling), or new data arrive from a server only the changed parts rerender, without even using special component, they typically lack some required feature. Or, imagine, it is a game scene rendering where some object changes but the code doesn't bother on handling the changes, it just renders the whole scene once, the rest is handled by the underlying framework.

I believe it is fairly easy to utilize Mobx or derivablejs for this. From EffectfulJS it will receive proper continuations (which may be fired more than once). And they are used called the same way @computed properties are called on dependency change. There is already delimited continuations effects library to be utilizied, only dataflow dependency tracking engines must be integrated.

A couple of links about Adaptive Programming (though they are about functional programming)

Adaptive Functional Programming
Monads for Incremental Computing
Traceable Data Types for Self-Adjusting Computation - mobx supports only adaptive references, this describes how to extend to more advanced types, like sorted maps.
This is based on rather complex functional programming concepts but end-user won't need to know them to use. They will just write plain JS programs.

• Adaptive Functional Programming
• Monads for Incremental Computing
• Traceable Data Types for Self-Adjusting Computation - mobx supports only adaptive references, this describes how to extend to more advanced types, like sorted maps.

This is based on rather complex functional programming concepts but end-user won't need to know them to use. They will just write plain JS programs.

corejs integration

The debugger requires everything including runtime to be instrumented. The current version has a few runtime functions implemented but the number is incomplete and its correctness isn't verified.

This is a big task to write from scratch but there is an open-source library - corejs. It can be instrumented (in blackbox mode) and loaded into the process, monkey-patching all the default implementations.

Breakpoints sometimes not work

Hi,

I have a problem when I use Effectful Javascript Debugger extension in vscode, hope I did not go to the wrong place.

In the codes below, when a breakpoint set on line 'console.log(line);', assume two lines in input.txt, so the breakpoint line should be stopped twice when debug run, but actually only stopped when first line read.

var fs = require('fs'),
    readline = require('readline');

var rd = readline.createInterface({
    input: fs.createReadStream('path/to/input.txt'),
    output: process.stdout,
});

rd.on('line', function(line) {
    console.log(line);
});

The launch config is created by default:

{
	"type": "effectful",
	"request": "launch",
	"name": "Launch Node application",
	"preset": "node",
	"cwd": "${workspaceRoot}",
	"command": "node",
	"args": [
		"${file}"
	],
	"console": "integratedTerminal",
	"env": {}
}

The version of the Effectful Javascript Debugger extension I installed is v1.3.42

Reactive programming library

Port old interface rx library to, maybe, proposed standard Observables or some other FRP library.

WebWorker data URLs

instrument data URLs for WebWorkers