revelrylabs / elixir-nodejs

An Elixir API for calling Node.js functions

NodeJS

Provides an Elixir API for calling Node.js functions.

Documentation

The docs can be found at https://hexdocs.pm/nodejs.

Prerequisites

• Elixir >= 1.7
• NodeJS >= 10

Installation

def deps do
  [
    {:nodejs, "~> 2.0"}
  ]
end

Starting the service

Add NodeJS to your Supervisor as a child, pointing the required path option at the directory containing your JavaScript modules.

supervisor(NodeJS, [[path: "/node_app_root", pool_size: 4]])

Calling JavaScript module functions with NodeJS.call(module, args \\ []).

If the module exports a function directly, like this:

module.exports = (x) => x

You can call it like this:

NodeJS.call("echo", ["hello"]) #=> {:ok, "hello"}

There is also a call! form that throws on error instead of returning a tuple:

NodeJS.call!("echo", ["hello"]) #=> "hello"

If the module exports an object with named functions like:

exports.add = (a, b) => a + b
exports.sub = (a, b) => a - b

You can call them like this:

NodeJS.call({"math", :add}, [1, 2]) # => {:ok, 3}
NodeJS.call({"math", :sub}, [1, 2]) # => {:ok, -1}

In order to cope with Unicode character it is necessary to specify the binary option:

NodeJS.call("echo", ["’"], binary: true) # => {:ok, "’"}

There Are Rules & Limitations (Unfortunately)

• Function arguments must be serializable to JSON.
• Return values must be serializable to JSON. (Objects with circular references will definitely fail.)
• Modules must be requested relative to the path that was given to the Supervisor. E.g., for a path of /node_app_root and a file /node_app_root/foo/index.js your module request should be for "foo/index.js" or "foo/index" or "foo".

Running the tests

Since the test suite requires npm dependencies before you can run the tests you will first need to run

cd test/js && npm install && cd ../..

After that you should be able to run

mix test

Handling Callbacks and Promises

You can see examples of using promises in the tests here:

https://github.com/revelrylabs/elixir-nodejs/blob/master/test/nodejs_test.exs#L125

and from the JavaScript code here:

module.exports = async function echo(x, delay = 1000) {
  return new Promise((resolve) => setTimeout(() => resolve(x), delay))
}

https://github.com/revelrylabs/elixir-nodejs/blob/master/test/js/slow-async-echo.js

elixir-nodejs's Issues

Async function call performance

I wrote a JS function that fetches some data and uses it to render a React/Vue component. I decided to do some load testing and for consistency's sake I mocked the data call with a promise that returns after 40ms.

50 virtual users sending 5 requests each:

artillery quick --count 50 -n 5 http://localhost:4000/render

Results with a pool size of 4:

Summary report @ 02:09:01(-0500) 2019-05-16
Scenarios launched: 50
Scenarios completed: 50
Requests completed: 250
RPS sent: 84.46
Request latency:
min: 44.3
max: 512.7
median: 357.2
p95: 483.4
p99: 503.2
Scenario counts:
0: 50 (100%)
Codes:
200: 250

I haven't worked much with OTP yet but it seems like each worker needs to wait for the async request to resolve. The same behavior can be observed in test "gets resolved value". Calling NodeJS.call("slow-async-echo", [1234]) a second time will add another second to the test execution time.

@bryanjos Any way around this? I'd be happy to take a stab at a PR or help in whatever way. Figured I'd inquire if anything jumps out to you.

Generators

I'm curious as to whether it's possible, and what it would take to support ES6 (async) generators.

I imagine we could support them through the stream APIs

Implementation leaks still there (2.0.0)

Oh no, I made it worse 😱

It looks like my PR that was supposed to get rid of implementation leaks introduced some new ones.
I should have tested my changes thoroughly with our production code...

It looks like my GenServer that calls NodeJS.call(..) is sometimes getting back the messages sent by Port:

{:ok, [message: {#Port<0.2131>, {:data, {:eol, []}}}, module: NodeJS.Worker, name: #PID<0.1656.0>]}

Also, I'm still getting the timeout messages in my mailbox:

** (FunctionClauseError) no function clause matching in CallerGenServer.handle_info/2
    (foo 1.1.0) lib/foo/callerGenServer.ex:91: CallerGenServer.handle_info({#Reference<0.221168823.2908225537.225316>, {:error, :timeout}}, nil)
    (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
    (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
    (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3

---

I suspect some race condition with timeouts...

Also, we might need to add a wrapper GenServer that catches all these messages such that the caller is truly isolated from the worker process.

---

Unfortunately I wont have much time to investigate exactly when and how it happens, as I'm soon leaving for vacations.
Otherwise I would solve it myself and contribute the solution.
I feel obliged to fix it, but I won't have the time (earliest after ca. 4 weeks).

I'm terribly sorry about that.

Using elixir-nodejs

Hi, I am having difficulty configuring elixir-nodeJS.

Here is what, I did in Application.ex.

%{
id: NodeJS,
start: {NodeJS, :start_link, [[path: "/workspace/lib/keila/JS", pool_size: 4]]}
}

In demos.ex

defmodule Keila.Demos do
import NodeJS
def add_and_print do
result = NodeJS.call("echo", ["hello"]) #=> {:ok, "hello"}
IO.puts "The sum is #{result}"
end
end

But the error is coming as:

iex(1)> c("demos.ex")

== Compilation error in file demos.ex ==
** (CompileError) demos.ex:2: module NodeJS is not loaded and could not be found

** (CompileError) compile error
(iex 1.14.4) lib/iex/helpers.ex:204: IEx.Helpers.c/2
iex:1: (file)

JS function calls fail when returning JSON is larger than 65536 bytes

As far as I understand this is related to maximum Port buffer size but I wasn't able to find the relevant Erlang documentation.

Error example:

         ** (EXIT) an exception was raised:
             ** (Jason.DecodeError) unexpected end of input at position 65536
                 (jason) lib/jason.ex:78: Jason.decode!/2
                 (nodejs) lib/nodejs/worker.ex:39: NodeJS.Worker.decode/1
                 (nodejs) lib/nodejs/worker.ex:32: NodeJS.Worker.handle_call/3
                 (stdlib) gen_server.erl:661: :gen_server.try_handle_call/4
                 (stdlib) gen_server.erl:690: :gen_server.handle_msg/6
                 (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

Call function inside ESM module

Hi 👋

I'm working on implementing Vue.js SSR for Phoenix.

To bundle server code, I'm using Vite. Sadly, it only generated ESM modules (using import / export, not require).

Currently, elixir-nodejs always require files, thus making it impossible to call functions inside ESM modules.

Solution

I'll implement import inside server.js and let it be used by specifying esm: true in opts, like this:

# extension is required in ESM imports
NodeJS.call({"esm-module.js", :uuid}, [], esm: true)

# if using mjs extension esm: true will be automatically set
NodeJS.call({"esm-module.mjs", :uuid})

# to force module reload, add a cache busting string
NodeJS.call({"esm-module.mjs?q=#{System.unique_integer()}", :uuid})

Caveats

• Extension is always required in ESM imports, if not importing package from node_modules.
• NODE_PATH is not respected by imports. It would need to be manually implemented
• there is no cache that could be busted between imports. There's a hack of adding ?update=RANDOM query string to import, but I believe it should be caller responsibility to do it, if necessary.

remove

remove

NODE_PATH must use semicolons on Windows

elixir-nodejs/lib/nodejs/worker.ex

Line 29 in 11151e1

|> Enum.join(":")

remove

Strange characters aren't transmitted properly

As an example, echoing ’ returns <<195, 162, 194, 128, 194, 153>> instead of ’.

Handling callbacks and promises

Hey,

How does elixir-nodejs handle function callbacks and promises? I couldn't find those topics on docs.

Cheers,
Gustavo

Broken encoding

Hi!

I have been trying the library (which is fantastic!) but I'm experiencing issues with characters encoding. If the return string from the called JS function contains UTF-8 characters, the return on Elixir side will be a binary.

Please, allow me to illustrate with en example with a letter "Ł":

Javascript:

module.exports = function (text){
    return text
}

Elixir:

iex> {:ok, x} = NodeJS.call("test",["hełło"])
iex> String.codepoints(x)
["h", "e", "Å", <<194, 130>>, "Å", <<194, 130>>, "o"]

Thank you!

Can Node.js keep state?

So this library keeps a set of workers in a pool to interface with Node.js. If I want to execute some Node.js code, is it possible to have an ongoing session or state? That is, if I make the same function call more than once I might get a different result back the second or third time? And, if so, would I need to have a "sticky session" with the user to make sure I get the same ongoing state?

I could write some JS code to store things or disk or in a DB and I might just do that, but I wanted to check first what this library is capable of. Thanks! 😊