rlugojr / nodeapp

[IN PROGRESS, STAY AWAY]

NodeApp

Use NodeApp to write desktop apps in a mix of Node.js and native code.

The native (“frontend”) code handles the view part of MVC, while the Node.js (“backend”) code handles the controller and model parts. Uses native Cocoa on a Mac, and planning to use Qt on Windows and Linux.

NodeApp contains:

• an RPC library to communicate between the native and Node.js sides via JSON messages — if you ever used postMessage or modern browser extensions API, you should be familiar with the style;

• a native ‘core’ that initializes the app and transfers control to the RPC library to launch the backend and start processing its requests;

• a few optional native modules that add certain RPC endpoints (UI Elements library, FS monitoring, native preferences access, license management);

• a reactive depedency tracking library for the Node side;

• a UI Elements architecture that drives the binding between the native controls and the Node controllers (implemented as a module on the native side and as a library on Node side).

Status

NodeApp is my cross-platform strategy for the LiveReload app. It is a work in progress and will mature together with LiveReload 3, due to be released in summer 2012.

Currently NodeApp lives in the nodeapp folder of the LiveReload repository. There are no skeleton apps provided yet. As soon as the framework is remotely ready for outside consumption, I will extract it into a separate repository.

RPC

RPC overview

Starts up, monitors and talks to the backend. Currently the backend runs in a separate process and speaks JSON on stdin/stdout; in the future I might try to embed Node.js as a separate thread of the main process.

(I do love being able to run the backend manually from the command line, feed some JSON into it and get responses in return. That feels like the Unix way, so I don't think I will ever give up this ability completely.)

The native side of NodeApp would happily talk to Python or Ruby backends just as well, if someone would write the code to receive and send the same JSON payloads on stdin/stdout. I'm only interested in Node.js currently, though, so that's the only existing implementation.

Status: The previous version of this code powers the currently shipping versions of LiveReload on Mac and Windows platforms, so it is known to work well. The code has been reorganized to be a part of NodeApp, though, and the new version has only been used for development so far.

Source code:

core/nodeapp.h
core/nodeapp_private.h
core/nodeapp_rpc_*.h
core/nodeapp_rpc_*.c
core/{mac,win}/nodeapp_rpc_osdep.c

RPC protocol

RPC subsystem sends and receives JSON-encoded messages of the following format:

["some.command", { "key1": "value1" }]

Each stdin and stdout line that starts with a square bracket is a message. The entire message must be a single line and must be a valid JSON array with 2 or 3 elements:

• a command name (a string),
• an arbitrary argument (of any type), should be null if unused,
• optional callback command name (a string), only supported for backend-to-native messages.

RPC protocol callbacks

Commands sent by the backend to the native side may specify an optional callback as the third element of the command array:

// backend to native
["some.command", { "key1": "value1" }, "$42"]

When the command is finished, the callback will be sent back to the backend as a command:

// native to backend
["$42", null]

An arbitrary value can be returned that way, too:

["$42", { "foo": ["bar", "boz", 123, {"fubar": "booboz"}] }]

The callback can only be used once — after $42 is invoked, the callback id may be reused.

Additionally, permanent callbacks can be specified inside the argument object (of backend-to-native messages only). For example, we could have a command like this:

// backend to native
["imaginaryui.bindEventListeners", { "to": "someControlId", "onclick": "$14", "ondrag": "$15" }]

In this case, the callback can be invoked multiple times:

// native to backend
["$15", { "x": 456, "y": 789 }]
["$15", { "x": 567, "y": 890 }]

and must be explicitly destroyed by the native side when no longer needed:

// native to backend
["-$15", null]

Sending and receiving RPC messages on the native side

To send a message, you invoke a function generated by rake routing:

json_t *arg = json_object();
json_object_set_new(arg, "key1", json_string("value1"));
json_object_set_new(arg, "key2", json_integer(42));
S_some__command(arg);

Note that the ownership of the arg is transferred to the proxy function, which will invoke json_decref after the command is sent.

To receive a message, you need to declare a function like:

void C_some__command(json_t *arg) {
    printf("key1 = %s\n", json_string_value(json_object_get(arg, "key1")));
}

These functions will be found by rake routing and put into a routing table used by the RPC.

Camel case message names are converted to an underscore style for the native side, so you use S_some_module__some_method to call someModule.someMethod, and define C_some_module__some_method to handle someModule.someMethod.

Callbacks. You can return a value to the callback provided by the backend if you declare your function as returning json_t *:

json_t *C_some__command(json_t *arg) {
    char *buf[102400];
    sprintf(buf, "key1 = %s\n", json_string_value(json_object_get(arg, "key1")));
    return json_string(buf);
}

Note that if you return json_t *, the backend side must provide a callback (or an assertion failure will occur), and currently there is no way to return the value asynchronously.

Permanent callbacks. You can invoke and/or dispose permanent callbacks using these functions (declared in nodeapp.h):

void nodeapp_rpc_invoke_and_keep_callback(const char *callback, json_t *arg);
void nodeapp_rpc_invoke_and_dispose_callback(const char *callback, json_t *arg);
void nodeapp_rpc_dispose_callback(const char *callback);

Please keep in mind that to store the callback names, you need to either json_incref the incoming argument or strdup the names, because the arguments (including any embedded strings) are deallocated when the handler returns.

Additionally, on a Mac, you can use these pure Objective-C alternatives:

void NodeAppRpcInvokeAndKeepCallback(NSString *callback, id arg);
void NodeAppRpcInvokeAndDisposeCallback(NSString *callback, id arg);
void NodeAppRpcDisposeCallback(NSString *callback);

(These alternatives are handy because Objective-C blocks automatically handle reference counting for Objective-C objects, but not for JSON objects. Also UI Elements lib supports routing requests to pure Objective-C implementations written in terms of NSString, NSDictionary and NSArray instead of pure C and JSON data types. The RPC lib does not currently support that, but might do so in the future.)

Sending and receiving RPC messages on the backend side

To send a message, you invoke a function on a global object called C, providing an argument and an optional callback:

C.someModule.someMethod { key1: "value1"}, (err, value) ->
  throw err if err
  console.log "Received %j", value

Incoming messages are currently routed by the application code like this:

_api =
  someModule:
    someMethod: (arg, callback) =
      # ...process...
      callback(null)

_rpc.on 'command', (command, arg, callback) ->
  try
    get(_api, command)(arg, callback)
  catch err
    callback(err)

Any functions in the outgoing messages are turned into permanent callbacks:

C.imaginaryui.bindEventListeners
  to: "someControlId"
  onclick: ->
    console.log "clicked"
  ondrag: ({x, y})->
    console.log "dragging at (#{x}, #{y})"

Note that the native side must dispose such callbacks properly, otherwise they will be leaked.

Reactive dependency tracking

The reactive library reruns code blocks when their dependencies change (and discovers those dependencies automatically):

R = require('reactive')
R.run "compute some prop", ->
  someModel.z = someOtherModel.x + yetAnotherModel.y

The code block in question is invoked automatically. Additionally, if it accesses any reactive properties, it will start listening to those properties and will be re-run when the properties change.

The name provided to R.run is used for debugging purposes only.

To declare an object with reactive properties:

class Project extends R.Entity
  constructor: ->
    super("some descriptive name")  # optional, name used for debugging only
    # this.__uid is set to a unique name by R.Entity constructor

    @__defprop 'x', 42         # 42 is the initial value
    @__defprop 'y', 100
    @__defprop 'z', 200

You can use these properties as usual, but they will report change events to dependent R.run blocks:

project1 = new Project()
project1.x = 10  # as usual

R.run "report stuff to console", -> console.log "project1.x = #{project1.x}"
# 10 is printed to the console

project1.x = 11  # 11 is printed
project1.x = 12  # 12 is printed
project1.x = 12  # nothing is printed because x hasn't changed

Naturally, R.run blocks can assign values to properties that other blocks depend on:

R.run "compute project1.x", -> project1.x = project1.y + project1.z
# now project1.x is set to 300, and 300 is printed to the console too

project1.y = 500
# now project1.x is 700; 700 is printed to the console

project1.z = 0
# now project1.x is 500; 700 is printed to the console

As a syntax sugar, any method of R.Entity that starts with automatically_ (where _ is a space) is convereted into a block:

class Project extends R.Entity
  constructor: (@name) ->
    super(@name)
    @__defprop 'x', undefined
    @__defprop 'y', 42

  'automatically compute x': ->
    @x = @y * 2

  'automatically print y': ->
    console.log "#{@name}.y = #{@y}"

p1 = new Project("p1")
p1.y = 10

p2 = new Project("p2")
p2.y = 20

If you enable debugging by setting env variable LOG to something like nodeappjs.reactive:pretty:stdout, you will see a trace of invocations like p1_compute_x and p2_print_y. We're using dreamlog library for logging; unfortunately, it is not documented yet.

A block and a property can be combined into a derived property:

class Project extends R.Entity
  constructor: (@name) ->
    super(@name)
    @__defprop 'y', 42
    @__deriveprop 'x', => @y * 2

This does the same thing as the previous example, but additionally disallows any (other) assignments to x.

Status: Seems to work, but not tested in production and still undergoes significant changes from time to time.

UI Elements

UI Elements overview

UI Elements are implemented on top of the RPC subsystem as ui.update native-side endpoint and ui.notify backend-side endpoint. UI Elements enable backend controllers to communicate with native widgets so that:

• the backend code is pleasant and artful,
• the backend code is easily inspectable and testable,
• there's no boilerplate code anywhere,
• you can drop back to native code at any point to handle stuff that would be harder to do in JavaScript.

We could handle the UI using a large number of RPC methods like ui.button.setTitle and ui.addEventListener (like in the examples above), but:

• RPC calls are a mess when it comes to testing
• we're exposing a fundamentally object-based API here, and doing that via RPC calls is downright ugly

So instead, we represent the application UI as a bunch of elements sitting in a tree, that is, as a tree of UI elements such as windows, controls, tree items and menu items. Native and backend sides then exchange parts of this tree as updates/notifications.

Status: Works on Cocoa with some elements missing (notably, there's no support for menu items yet) and will likely continue to change in the future.

UI Elements JSON payload examples

For example, the native ui.update RPC endpoint could be invoked by the backend with an argument like:

'#mainwindow':
  type: 'MainWindow'
  visible: true

  '#addProjectButton': {}
  '#removeProjectButton': {}

  '#projectOutlineView':
    style: 'source-list'
    'dnd-drop-types': ['file']
    'dnd-drag': yes
    'cell-type': 'ImageAndTextCell'
    data:
      '#root':
        children: ['#folders']
      '#folders':
        label: "MONITORED FOLDERS"
        'is-group': yes
        children: ['#folder1', '#folder2']
        expanded: yes
      '#folder1':
        label: "~/Foo/Bar"
        image: 'folder'
        expandable: no
      '#folder2':
        label: "/some/dir"
        image: 'folder'
        expandable: no

  '#gettingStartedView':
    visible: no

And the backend ui.notify RPC endpoint could be invoked with an argument like:

"#mainwindow": {
  "#projectList": {
    "selected": "#folder2"
  }
}

UI Element Tree details

Every UI element has a string ID starting with a hash mark; unlike HTML, string IDs are only required to be unique within their parent.

Some elements are assigned their IDs at compilation time; for example, Cocoa views specified in a xib file and assigned to the outlets of a custom window controller use their outlet names as IDs: an outlet named addProjectButton is accessible as #addProjectButton.

Other IDs can be arbitrarily assigned by the backend if enough information is provided to create the corresponding elements. For example, the native side does not know about #mainwindow ID initially; however, because the backend tells that it has a "type": "MainWindow", the native UI lib knows how to create the window in a platform-specific way — the Cocoa implementation will look for MainWindow.xib and MainWindowController class here.

Native side hooks up and reports events for the controls that have been mentioned in JSON updates at least once. That's why "#removeProjectButton": {} is specified.

To delete a UI element, "#something": false can be used.

UI Element controllers

The Node UI Elements library allows you to write UI code inside controllers, adding a special $ method for sending updates, and using declarative CSS-like syntax to bind incoming notifications to their handlers:

class ApplicationController

  initialize: ->
    @$ '#mainwindow':
      type: 'MainWindow'
      ...

  '#mainwindow #addProjectButton clicked': ->
    @$ '#mainwindow': '#statusTextField': text: "Clicked the Add Project button!"

The parallel to HTML and CSS (including the hash marks) may seem random, however we do have very similar case here and many CSS and jQuery concepts can be applied. In particular, with support something very similar to CSS classes, so that you could bind a handler to #mainwindow .someButtonClass clicked. (We could do away without hash marks, but they are very useful for grepping.)

UI Element controllers hierarachy

Handling the entire tree on the Node style would still be a mess, so the UI library allows you to define a hierarchy controllers, each controller bound to a certain element of the tree.

For example, this ApplicationController basically delegates everything to a main window controller:

class ApplicationController

  initialize: ->
    @$ '#mainwindow': {}

  '#mainwindow controller?': -> new MainWindowController

The main window controller handles some stuff itself, but employs a subcontroller for managing the list of projects. Note how the controller doubles as a presentation model; conceptually, those are one and the same:

class MainWindowController extends R.Entity

  constructor: ->
    @__defprop 'selectedProject', null
    @__defprop 'visiblePane', 'welcome'

  initialize: ->
    @$ visible: true

  render: ->
    @$ '#welcomePane': visible: (@visiblePane is 'welcome')
    @$ '#projectPane': visible: (@visiblePane is 'details')

  '%projectList controller?': -> new MainWindowProjectListController(this)

And the project list controller handles the tree view plus the add/remove project buttons:

class MainWindowProjectListController

  constructor: (@model) ->

  initialize: ->
    @$
      '#projectOutlineView': {}
      '#gettingStartedView': visible: no

  '#projectOutlineView selected': (arg) ->
    if project = (arg && LR.model.workspace.findById(arg.substr(1)))
      @model.selectedProject = project

  render: ->
    @$ '#projectOutlineView': 'data': convertForestToBushes [
      id: '#folders'
      children:
        for project in LR.model.workspace.projects
          id:    "#" + project.id
          label: project.name
          tags:  '.project'
    ]

  ...

(Don't ask me about the question mark in controller? — that was just a stupid internal decision. Also don't ask me about convertForestToBushes, that's shenanigans of the Outline View element.)

The initialize method is called once at the start; render method is called afterwards within an R.run block, so it will be called again and again to re-render the UI when the model changes. You can also use automatically something methods like in R.Entity subclasses (in fact, some of your controllers will be R.Entity subclasses, as the example shows).

UI Element data binding

Some UI elements are naturally handled as a render method and event handler methods. In simple cases, however, that may get boring:

class MonitoringOptionsController

  constructor: (@project) ->

  initialize: ->
    @$ visible: yes

  render:
    @$ '#disableLiveRefreshCheckBox': state: @project.disableLiveRefresh

  '#disableLiveRefreshCheckBox clicked': (newState) ->
    @project.disableLiveRefresh = newState

(Repeat for all 5 checkboxes in that window.)

That's why the UI library also supports declarative data binding:

class MonitoringOptionsController

  constructor: (@project) ->

  initialize: ->
    @$ visible: yes

  '#disableLiveRefreshCheckBox checkbox-binding': -> @project.disableLiveRefresh$$

The magic disableLiveRefresh$$ property is an object with get and set methods which get or set the value of the corresponding disableLiveRefresh property. (Both are created by the __defprop call in the R.Entity subclass.)

UI Elements stylesheet

Similar to how CSS allows you to extract presentation information from HTML and JavaScript code, UI Elements library also supports stylesheets (which are merged with outgoing UI updates, right before sending them from the backend side).

Currently the styles are specified as JSON, although we should probably add a layer that reads a real CSS file (compiled from something like Stylus, LESS or Sass) and converts that to JSON:

module.exports =

  '#mainwindow':
    'type': 'MainWindow'

  '#mainwindow #projectOutlineView':
    'style':           'source-list'
    'dnd-drop-types': ['file']
    'dnd-drag':         yes
    'cell-type':       'ImageAndTextCell'

  '#mainwindow #snippetLabelField':
    'hyperlink-url': "http://help.livereload.com/kb/general-use/browser-extensions"
    'hyperlink-color': "#000a89"
    'cell-background-style': 'raised'

  ...

This allows to provide properties that will not change (like type) and also add platform-specific styles and behaviors that cannot be set in Interface Builder (like cell-background-style or dnd-drop-types).

Naturally, we are going to have platform-indendepent stylesheets and platform-specific stylesheets. (Could even reuse @media for that, although right now @media seems like an overkill.)

Starting a NodeApp project

You need to have Ruby and Node.js installed. Ruby 1.8.7 which comes with OS X works fine; haven't tried with Ruby 1.9. Tested with Node 1.6; won't work with 1.4, might work with later versions.

Here's the recipe for a project:

• Make a copy of the skeleton app

• Customize app_config.h (provided by the skeleton app)

• Set the current version number in mac/Info.plist and run rake ver:mac:update to update the version number everywhere (currently the only other copy is in app_version.h, see the definition of MacVersion in the Rakefile)

• Run rake prepare to install all prerequisites and compile the backend

• Run rake routing to generate shared/gen_src/nodeapp_rpc_proxy.h, shared/gen_src/nodeapp_rpc_proxy.c (which provide function stubs for all RPC endpoints exposed by the Node app) and shared/gen_src/nodeapp_rpc_router.c (which routes the incoming RPC requests to their respective native implementations). This task starts up the backend to query the list of RPC endpoints, so it will fail if the backend fails to run.

• Open mac/YourProject.xcodeproj in Xcode, build and run. (Qt part will have something similar.)

• Write your app (see a section on that below)

• Shake well

• Distribute

What goes where

• shared/src: native sources shared between the platforms (app_config.h lives here)

• shared/gen_src: files generated by rake routing

• {mac/win}/src/app_version.h: version number is defined here (could be different across the platforms)

• {mac,win}/src/vendor: third-party libraries, fragments of Apple/MSDN examples and code snippets hunted down on StackOverflow

• mac/src/ui: your xibs and NSWindowController descendants

• mac/src/app/AppDelegate.m: lifecycle events of your app, if you need to run some custom code

• mac/src/main.m: execution actually starts here

• mac/Info.plist: defines everything Apple and OS X want to know about your app

• mac/App.icns: your icon

• mac/MainMenu.xib: duh, define your menu structure here

Writing an app

• start with a skeleton app as described above

• build your views layer using the native techologies: create an empty NSWindowController subclass for each window, design the xib in Interface Builder; not exactly sure about the Qt part yet — I'm thinking about auto-converting xibs into Qt UIs

• build your controllers layer in Node.js using the UI Elements and (optionally) the reactive dependency tracking

• build your model layer in Node.js, optionally using the reactive dependency tracking library

• add the necessary bits of native code to handle stuff that you cannot or do not want to do in JavaScript

Creating the skeleton app from scratch

1. Put NodeApp into nodeapp/ subfolder of your repository. Preferably, add it as a submodule.

2. Add ‘nodeapp’ folder to Xcode (don't “copy items into destination group's folder”, do “create groups for any added folders”). Remove:

    nodeapp/*/src-win
    nodeapp/fsmonitor/libs/fsmonitor/{demo,**/*win32*}
   

3. Add app_config.h to your source files. (It should be shared between platforms.) Copy the contents from somewhere, adjust to your taste.

4. Add per-platform app_version.h file with a contents like this:

   #define NODEAPP_VERSION "2.5.0"

5. Change superclass of your app delegate to NodeAppDelegate. Change your applicationDidFinishLaunching: to call super. (If something needs to be initialized before the Node backend fires up, be sure to do it before calling super.)

6. Create a Rakefile with the following contents:

    #!/usr/bin/env ruby
    # -*- encoding: utf-8 -*-
   
    require 'rake/clean'
   
    Dir['tasks/*.rb'].each { |file| require file }
    Dir['nodeapp/*/tasks/*.rb'].each { |file| require file }
   
    MacVersion = VersionTasks.new('ver:mac', 'app/mac/Info.plist', %w(app/mac/src/app_version.h))
   
    RoutingTasks.new(
      :app_src        => 'LiveReload,Shared',
      :gen_src        => 'Shared/gen_src',
      :messages_json  => 'backend/config/client-messages.json',
      :api_dumper_js  => 'backend/bin/livereload-backend-print-apis.js'
    )
   

   (VersionTasks one is optional, and is used to keep your Info.plist in sync with your app_version.h. Be sure to specify your own paths if you use this line.)

7. Magically create a Node.js side of the RPC system, including that print-apis.js file mentioned on step 2.

8. Run rake routing.