38 / plumber

The General-Purpose Cross-Language Dataflow Programming

Plumber - The General-Purpose Cross-Language Dataflow Programming

---

What is Plumber?

Plumber is middleware for high-performance, general-purpose, cross-language dataflow programming. Plumber allows developer design dataflow based system easily, and provides many features, such as type-checking, generic-typing, metaprogramming, in a language-neutral way. For more details please visit the project home page.

Try Plumber

Try the Application - The PINS Web Server

The Plumber Project home page is actually powered by this web server. The code lives in the Plumber examples repository. You can play with the server in following ways.

• Try with Docker(Linux Only)

docker run --rm -t -i --network=host haohou/plumber-fileserver-example --port=8080

To serve the files other than the default page

docker run --rm --network=host -ti -v /path/to/serve:/www haohou/plumber-fileserver-example --root=/www

Explore the Framework with the Sandbox(Linux/MacOS/Windows WSL)

You can also use the sandbox environment and try the examples with the Plumber examples repository.

Tutorial

Now we have a tutorial repository in which we demonstrate how we build a simple server software setp by step. In this tutorial we are be able to go through most of the key concepts of Plumber software infrastructure. Follow the link for the tutorial repository.

Useful Links

• Plumber Project Site
• Plumber Main Project
• Plumber Examples Repository
• Introduction to Plumber (Slides)
• Explaination for the Plumber Based Static Content Server
• A Tutorial of Plumber

plumber's Issues

RLS Stream Cache

RLS Stream might be very computing intensive, for example, a RLS with pending operation for compression, encryption and transcoding. And this will make the framework unable to reuse the result of the computational intensive operations on some immutable objects.

For example, for the static content server, it actually loss 50% of the performance if we enable compression.

For the design details about this Cache, we actually need an RLS object that abstracts the cache.

Thus we also need a callback for the object hash code and if the function is not defined, indicates we just got a volatile object (For example an object that generates a random number)

Additionally, this hashing infrastructure can be used as ETag in the HTTP server.

Merge some of the servlet

Spliting too many servlet actually makes the server runs slow and graph be more complicated.
So we need to some of them, here's the candidate:

1. filesystem/mimetype and filesystem/readfile: Actually we could have one servlet which read the file and guess the mime-type, since there's no reason to split them.

2. network/http/encoder and network/http/render(Purposed): Since encoding a HTTP body has no other use than render a HTTP response.

Async Task Support

General Idea

We are planning to implement an asnyc task, which is the task do not need to hold the worker thread during running. This is meaningful when our servlet is supposed to perform some slow operation, like call other service and etc. Because the asyc task do not occupy the worker thread, so the same thread will be able to serve another request since then.

We need to limit what a async task can do, because if we don't have any limit on this, it probably introduce some race condition and break the lockless design. One resonable limit is, the asnyc task will only able to read data from the given buffer and do not have any acess to the framework related APIs, for example, pipe_read, etc.

So the async servlet should be defined as

    SERVLET_DEF = {
        .desc = "xxxxxx";
        .init = _init,
        .exec_init = exec_init,   /* The function will be called by worker thread to setup the async task */
        .exec_async = exec_async, /* The function will be called by async worker to do some slow ops */
        .exec_finalize = exec_final,  /* Clean up function called by worker thread */
        .unload = _unload
    }

Detailed Process

• The scheduler picked up an async task
• In sched_task, we need add a table keep tracking the pending asnyc tasks.
  When the async task has been picked up, the exec_init call will be called, then the exec_async will be called by an async worker.
• We need to implement a async worker pool and async thread which can accept a async task context from the scheduler. The async woker should be another from of even source and occupy a event loop token.
• Add another type of event, (EVENT_TYPE_MODULE and EVENT_TYPE_TASK)
• After the async task is done, the async loop should post a task event to the event queue
• The dispatcher picked up the task event, and instead of sending it to a random worker thread, we need send it to the specified worker. (Which means we need keep tracking the scheduler context in the event)
• Finally, the worker scheduler picked up the event, and should remove the async task specified by the event from the pending async task queue and call exec_finalize function, then dispose the task.

Issue

Currently in the sched_step_next function, we mark the pipe ready right after the upstream output has been created. But this is no longer ture, and we should prevent the downstream task from starting until the async task is finished.
This prbably can be done by not calling the sched_task_input_pipe right after the pipe pair has been created. Instead, we call the sched_task_input_pipe when we are going to run exec_finalize.

Full support for container types

Currently, we have merged the container RLS types, but is far away from finished.

pstd_blob_* is the function that manages a typed memory region. It also caches the RLS object for future use.

pstd_array_* is the implementation for the array type. (Needs the blob model to create)

pstd_type_model_field_t served as the accessor in a memory blob

For the generic servlet with type "Array $T", we need add a type assertion and in the type assertion
we will be able to create the blob model, since $T is known.

For each time we want to read/write a RLS token using the cache, we need to use the blob model.

But for now, it's better to add some laziness to the HTTP parsere. (For exmaple, some unused header never gets parsed, etc.) So it seems we need to implement a seperate library so that we can create some RLS object specific for HTTP senario rather than use the generic one

Pscript interactive client

• help() function
• sigint handler
• Output stdout or stderr instead of log

I will create pull request about these.

Support the Async Task and Complete the RESTful storage servlet

It seems we actually don't want use the filesystem as a RESTful storage, because the data consistancy issues.

However, once we create a storage servlet wraps another data storage program, like mogodb, etc. Then we actually need the async task model to make it has high troughput.

This makes the task not feasible at this point, because the async task is not supported by the framework.

At the same time, it seems the async task support should have the highest priority, because we need this for:

1. Reverse proxy application
2. Restful Storage
3. Multiple Instance Deployment

So we need to do the RESTful after we finished the Async Task Support

Automated test for servlet by adding json servlet