suprematic / otplike

Erlang/OTP like processes and behaviours for Clojure on top of core.async

otplike

otplike is a framework built on top of core.async. It emulates basic Erlang/OTP concepts, such as processes, process linking, monitoring, standard behaviours.

Rationale

Although core.async provides a solid foundation for asynchronous applications, our experience shows that there is a need in higher level system building blocks.

It appears that certain ideas can be taken from Erlang/OTP and implemented on top of core.async.

The gen_server equivalent is used to serialize sync/async access to state and ensure that possibly inconsistent state data will be discarded in case of a crash.

Process linking/monitoring improves crash/error propagation and supervision covers recovery. In addition process tracing facility helps a lot with application debugging and profiling.

It is obvious that due to JVM limitations otplike cannot replace Erlang/OTP and otplike will NEVER be seen as Erlang/OTP alternative.

Example

Echo Server

(require '[otplike.process :as process :refer [!]])

(process/proc-defn server []
  (println "server: waiting for messages...")
  ; wait for messages
  (process/receive!
    [from msg]
    (do
      (println "server: got" msg)
      ; send response
      (! from [(process/self) msg])
      (recur))
    :stop
    ; exit receive loop
    (println "server: stopped")))

(process/proc-defn client []
  ; spawn process
  (let [pid (process/spawn server)]
    ; send message to it
    (! pid [(process/self) :hello])

    ;wait for response
    (process/receive!
      [pid msg]
      (println "client: got" msg))

    ; ask spawned process to stop
    (! pid :stop)))

(process/spawn client)

More examples are available under the /examples directory.

Releases and Dependency Information

All Released Versions

Leiningen dependency information:

[otplike "0.6.0-alpha"]

Documentation

• API docs
• Examples

Other materials

• EuroClojure 2017 presentation

Known issues

• A chain of N processes, when each next process is created by the previous one, holds amount of memory proportional to N until the last process' exit

Plans

• ClojureScript compatibility
• application behaviour and related features as configuration
• "Simple" supervisor (analogous to simple_one_for_one in Erlang) as a separate module
• Tracing and introspection
• More advanced examples/tutorial

Contributing

Please use the project's GitHub issues page for all questions, ideas, etc. Pull requests are welcome. See the project's GitHub contributors page for a list of contributors.

License

Copyright © 2017 SUPREMATIC and contributors.

Distributed under the Eclipse Public License v1.0, the same as Clojure. License file is available under the project root.

Changelog

• Release 0.6.0-alpha on 20.07.2019 NOTES
• Release 0.5.0-alpha on 07.12.2018 NOTES
• Release 0.4.0-alpha on 15.04.2018 NOTES
• Release 0.3.0-alpha on 28.11.2017 NOTES
• Release 0.2.1-alpha on 11.08.2017 NOTES
• Release 0.2.0-alpha on 17.05.2017 NOTES
• Release 0.1.0-SNAPSHOT on 15.08.2016

otplike's Issues

EuroClojure 2017 Slide 10 Possible Error

On slide 10 of the EuroClojure 2017 presentation the following code is displayed:

(proc-defn process2 []
  (receive!
    [:message m]
    (println message))
  (recur))

(proc-defn process1 []
  (let [p2 (process/spawn-link process2)]
    (receive 
      m
      (! p2 m))
    (recur)))

(let [p1 (process/spawn process1)]
  (! p1 :unsupported-message))

In "process1" is the line "(receive" supposed to be "(receive!"?

protocols for swapping out core.async and core.match?

I was wondering if you did (or are planning to) abstract these 2 core dependencies out? Then if a better implementation comes along, or one with different tradeoffs, it would be easy to mix&match them while preserving the otplike functionality.

As an example matching could be swapped for meander and core.async might find a contestant once loom comes out.

From a quick glimpse it seems the libraries are used directly

Require :reload-all Question

Given the following code, I can run "lein repl", everything compiles and runs without error. If I then execute:

(require 'otplike-log.core :reload-all)

I get the following error:

CompilerException java.lang.IllegalArgumentException: No implementation of method: :exec of protocol: #'clojure.core.async.impl.protocols/Executor found for class: clojure.core.async.impl.exec.threadpool$thread_pool_executor$reify__1518, compiling:(trace.clj:15:3)

The Code

(ns otplike-log.log
  (:require [clj-time.core :as t]
            [clj-time.local :as l]
            [otplike.process :as process :refer [!]]))

(defn pad0
  [field-width n]
  (let [s (pr-str n)
        pad-count (- field-width (count s))
        pad-string (apply str (repeat pad-count "0"))]
    (str pad-string s)))

(def pad00 (partial pad0 2))

(def pad000 (partial pad0 3))

(defn get-now-str
  []
  (let [now (l/local-now)]
    (str
      (pad00 (t/year now)) "-" (pad00 (t/month now)) "-" (pad00 (t/day now))
      " "
      (pad00 (t/hour now)) ":" (pad00 (t/minute now)) ":" (pad00 (t/second now))
      "." (pad000 (t/milli now)))))

(defn get-space-delimited-values-string
  [values]
  (apply str (interpose " " values)))

(defn get-log-line
  [values]
  (str (get-now-str) ": " (get-space-delimited-values-string values) "\n"))

(process/proc-defn
  log-proc
  [log-line-fn]
  (process/receive!
    msg (do (log-line-fn msg)
          (recur log-line-fn))))

(defn create-log
  [log-fn]
  (let [log (process/spawn log-proc [(comp log-fn get-log-line)])]
    (fn
      [& values]
      (! log values))))

and here's a demo of using it:

(ns otplike-log.core
  (:require [otplike-log.log :as l])
  (:gen-class))

(defn doit
  [this-many-times moniker sleep-ms log]
  (doseq [n (range this-many-times)]
    (do
      (log moniker n)
      (Thread/sleep sleep-ms))))

(def doit-slow (partial doit 10 "slow" 1000))

(def doit-fast (partial doit 20 "fast" 500))

(defn log-to-file
  [log-file-spec]
  (fn
    [log-line]
    (spit log-file-spec log-line :append true)))

(defn example-1
  [log]
  (log
    "one"
    (+ 1 2)
    (apply str (interpose "," "three")))
  (doseq [log-message (range 10)]
    (log log-message)))

(defn example-2
  [log]
  (future (doit-fast log))
  (doit-slow log))

(defn -main
  [& args]
  (let [log (l/create-log (log-to-file "log.txt"))]
    (example-1 log)
    (example-2 log)))

A default circuit-breaker implementation

A wrapper gen-server parameterized by the start function of the gen-server/supervisor to break failure of. The delay of the next start must also be somehow provided. Other parameters of the restart strategy are possible.

Remove inbox parameter of process function

As long process/receive! is here, there is no need to pass inbox to process function.
Also process' inbox should be available as process/*inbox*.

Event Module from http://learnyousomeerlang.com/designing-a-concurrent-application

The following is my first attempt at implementing the Event Module from learnyousomeerlang using otplike.

If you could take a look and provide feedback it would really help me make sure
I’m understanding how to use otplike properly.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; event.clj
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(ns otplike-event-example.event
  (:require [clj-time.coerce :as tc]
            [clj-time.core :as t]
            [clj-time.local :as tl]
            [otplike.process :as op]
            [otplike.proc-util :as opu]))

(defn get-to-go
  "Given a date-time - specified as vectors of integers for year, month and day,
  and hour, minute and second - return the number of milliseconds between now
  and the specified date-time.  If the specified date-time is before now return
  a zero."
  [[year month day] [hour minute second]]
  (let [target-date-time (t/date-time year month day hour minute second 0)
        local-target-date-time (tl/to-local-date-time target-date-time)
        local-target-date-time-millis (tc/to-long local-target-date-time)
        local-now-millis (tc/to-long (tl/local-now))
        to-go (- local-target-date-time-millis local-now-millis)]
    (if (< 0 to-go)
      to-go
      0)))

;; When the specified number of milliseconds 'to-go' elapses send a messge to
;; the specified 'server' process of the form '[:done event-name]'.  While
;; 'to-go' milliseconds has not elapsed, if a message is received from 'server'
;; of the form '[server ref :cancel]' send a message to 'server' of the form
;; '[ref :ok]' and exit the function.
(op/proc-defn
  event-loop
  [server event-name to-go]
  (op/receive!
    [server ref :cancel] (op/! server [ref :ok])
    (after to-go (op/! server [:done event-name]))))

(defn event-start
  "Given a 'server' process, 'event-name', and a time-date represented as two
  integer vectors of the form [year month day] and [hour minute second], start
  an event process named 'event-name' scheduled to send the 'server' a message
  of the form '[:done event-name]' when the time-date is reached.  If the
  time-date is now, or before now, 'server' will receive the message
  immediately.  At any time prior to reaching time-date if 'server' sends a
  message to the started event process of the form '[server ref :cancel]' the
  started event process will send a message to 'server' of the form '[ref :ok]'
  and will terminate execution.  Returns the event process."
  [server event-name year-month-day hour-minute-second]
  (op/spawn
    event-loop
    [server event-name (get-to-go year-month-day hour-minute-second)]))

(defn event-cancel
  "Send a cancel request from 'server' to an 'event' process.  If the event
  process is already canceled it will still work.  This function will not
  return until the event process is canceled.  Returns 'true'."
  [server event]
  (opu/execute-proc!!
    (let [monitor-ref (op/monitor event)]
      (op/! event [server monitor-ref :cancel])
      (op/receive!
        [monitor-ref :ok] (op/demonitor monitor-ref)
        [:DOWN monitor-ref _ _ _] nil)))
  true)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; event_test.clj
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(ns otplike-event-example.event-test
  (:require [clj-time.core :as t]
            [clj-time.local :as tl]
            [clojure.test :refer :all]
            [otplike.process :as op]
            [otplike.proc-util :as opu]
            [otplike-event-example.event :as oe]))

(defn get-now-plus-or-minus
  "Return a time-date instance for now plus or minus the specified 'seconds'."
  [seconds]
  (t/plus (tl/local-now) (t/seconds seconds)))

(defn get-y-m-d
  "Given a time-date instance return a vector of the form '[year month day]'."
  [td]
  [(t/year td) (t/month td) (t/day td)])

(defn get-h-m-s
  "Given a time-date instance return a vector of the form '[hour min sec]'."
  [td]
  [(t/hour td) (t/minute td) (t/second td)])

(defmacro do-receive
  ([done-name] (do-receive &form &env done-name true 0 false))
  ([done-name unique] (do-receive &form &env done-name false unique true))
  ([done-name done-is unique unique-is]
   `(~'op/receive!
      [~unique :ok]
      (~'is ~unique-is "received [ref :ok] message")

      [:done ~done-name]
      (~'is ~done-is "to-go timeout reached")

      :else
      (~'is false "unknown message")

      (~'after 2000 (~'is false "timeout occurred")))))

(deftest get-to-go-test
  (testing "target time after now"
    (let [later (get-now-plus-or-minus 10)]
      (is (< 0 (oe/get-to-go (get-y-m-d later) (get-h-m-s later))))))

  (testing "target time before now"
    (let [later (get-now-plus-or-minus -10)]
      (is (= 0 (oe/get-to-go (get-y-m-d later) (get-h-m-s later)))))))

(deftest event-loop-test
  (testing "to-go time is reached and to-go is greater than zero"
    (opu/execute-proc!!
      (let [name "test"
            server (op/self)]
        (op/spawn oe/event-loop [server name 1000])
        (do-receive name))))

  (testing "to-go time is reached and to-go is zero"
    (opu/execute-proc!!
      (let [name "test"
            server (op/self)]
        (op/spawn oe/event-loop [server name 0])
        (do-receive name))))

  (testing "[ref :ok] message received on cancel with small to-go"
    (opu/execute-proc!!
      (let [name "test"
            unique (gensym)
            server (op/self)
            event (op/spawn oe/event-loop [server name 1000])]
        (op/! event [server unique :cancel])
        (do-receive name unique))))

  (testing "[ref :ok] message received on cancel with very large to-go"
    (opu/execute-proc!!
      (let [years 5
            to-go (* years 365 24 60 60 1000)
            name "test"
            unique (gensym)
            server (op/self)
            event (op/spawn oe/event-loop [server name to-go])]
        (op/! event [server unique :cancel])
        (do-receive name unique)))))

(deftest event-start-test
  (testing "start one event process"
    (opu/execute-proc!!
      (let [server (op/self)
            name "test"
            later (get-now-plus-or-minus 1)
            y-m-d (get-y-m-d later)
            h-m-s (get-h-m-s later)]
        (oe/event-start server name y-m-d h-m-s)
        (do-receive name))))

  (testing "start multiple event processes"
    (opu/execute-proc!!
      (let [server (op/self)
            names #{"test-1" "test-2" "test-3"}
            later (get-now-plus-or-minus 1)
            y-m-d (get-y-m-d later)
            h-m-s (get-h-m-s later)]
        (doseq [name names]
          (oe/event-start server name y-m-d h-m-s))
        (loop [done-names #{}]
          (op/receive!
            [:done n]
            (let [names-so-far (conj done-names n)]
              (if (= names-so-far names)
                (is true "all events done")
                (recur names-so-far)))

            :else
            (is false "unknown message")

            (after 2000 (is false "timeout occurred"))))))))

(deftest event-cancel-test
  (testing "cancel running event process"
    (opu/execute-proc!!
      (let [server (op/self)
            name "test"
            later (get-now-plus-or-minus 1)
            y-m-d (get-y-m-d later)
            h-m-s (get-h-m-s later)
            event (oe/event-start server name y-m-d h-m-s)]
        (is (oe/event-cancel server event)))))

  (testing "cancel already canceled event process"
    (opu/execute-proc!!
      (let [server (op/self)
            unique (gensym)
            name "test"
            later (get-now-plus-or-minus 1)
            y-m-d (get-y-m-d later)
            h-m-s (get-h-m-s later)
            event (oe/event-start server name y-m-d h-m-s)]
        (op/! event [server unique :cancel])
        (do-receive name unique)
        (is (oe/event-cancel server event))))))

Add gen-server/stop fn

supervisor/which-children

Add an optional doc string to proc-defn

It would be nice to be able to do something like:

(proc-defn myproc
"This is my cool process function."
[p1 p1]
(receive!
.
.
.))

process/alive? function to check if process is alive

Example application for otplike with app restart feature

Just FYI...

We created a toy/example application for otplike:
https://github.com/ady1981/erl-like-app.clj

And article about it (in Russian): https://habr.com/post/354984/

Thanks!

Fix supervisor's :period property units

Either fix the supervisor's docs (changing seconds to milliseconds) or make :period to be in seconds.

Docstrings for all public process fns and macros

process/link call increases exit message delivery time

This test fails:

(deftest exit-self-reason-is-process-exit-reason
  (let [done (async/chan)
        done1 (async/chan)
        pfn (proc-fn []
              (await-completion done1 50)
              (process/exit (process/self) :abnormal))
        pid (process/spawn pfn [] {})
        pfn1 (proc-fn[]
               (process/link pid)
               (async/close! done1)
               (is (= [:exit [pid :abnormal]] (<! (await-message 50)))
                   "process exit reason must be the one passed to exit call")
               (async/close! done))]
    (process/spawn pfn1 [] {:link-to pid :flags {:trap-exit true}})
    (await-completion done 100)))

pfn1 receives :timeout instead of exit message.

And this test passes:

(deftest ^:parallel exit-self-reason-is-process-exit-reason
  (let [done (async/chan)
        done1 (async/chan)
        pfn (proc-fn []
              (await-completion done1 50)
              (process/exit (process/self) :abnormal))
        pid (process/spawn pfn [] {})
        pfn1 (proc-fn[]
               ;(process/link pid) <<< this line is commented out
               (async/close! done1)
               (is (= [:exit [pid :abnormal]] (<! (await-message 50)))
                   "process exit reason must be the one passed to exit call")
               (async/close! done))]
    (process/spawn pfn1 [] {:link-to pid :flags {:trap-exit true}})
    (await-completion done 100)))

Process exit reason may differ from process/exit argument

(process/spawn
  (proc-fn [] (process/exit (process/self) :abnormal)) [] {})

Exit reason of process spawned above may be either :normal or :abnormal and randomly changes between executions.

Close and flush process's inbox during (exit self reason) call

Remove internal symbols from project's API docs

Initial application module/behaviour

First of all, application is a convenient way to start/stop a root supervisor.

Next steps are:

• application configuration
• application dependencies

Add source links to API docs

spawn-opt doesn't throw exception when process name is already registered

process/spawn-opt called with :register option set to an already registered name fails silently.
Must throw an exception/exit.

Add proc-defn- macro as pair to proc-defn

It may be useful to define namespace-private process fns.

Questions: Process Function Matching

Consider a process function like:

(op/proc-defn
  event-loop
  [server event-name to-go]
  (op/receive!
    [server ref :cancel] (op/! server [ref :ok])
    (after to-go (op/! server [:done event-name])))) 

A couple of questions:

1. Will the match [server ref :cancel] only match messages from the specific value of server passed to event-loop, any value of ref and :cancel?
2. What happens to non-matching messages that might be sent to event-loop?
3. Should one do something to handle non-matching messages?

Selective receive

Add process/selective-receive! behaving the same way as receive in Erlang.

Function to get process' inbox outside if the process

Enhanced Example e4-receive-timeout

I attended the otplike talk at EuroClojure 2017. I have only just started to understand how I can use otplike.

I have successfully executed Example e4-receive-timeout. I also added additional output to e4-receive-timeout that helped me better understand the execution sequence of the example. For your consideration, my modified source code is attached.

e4_receive_timeout.clj.zip

Special processes running on the agents thread pool

Such processes are required to perform computationally intensive tasks or blocking calls.

Typo In Documentation Of otplike.process/exit

In the sentence "Throws when callen not in process context, if pid is not a pid, or reason is nil." the string "callen" should be "caller".

Example: otplike Logging System

Here's some code to implement a simple otplike logging system:

(ns otplike-log.log
  (:require [clj-time.core :as t]
            [clj-time.local :as l]
            [otplike.process :as process :refer [!]]))

(defn pad0
  [field-width n]
  (let [s (pr-str n)
        pad-count (- field-width (count s))
        pad-string (apply str (repeat pad-count "0"))]
    (str pad-string s)))

(def pad00 (partial pad0 2))

(def pad000 (partial pad0 3))

(defn get-now-str
  []
  (let [now (l/local-now)]
    (str
      (pad00 (t/year now)) "-" (pad00 (t/month now)) "-" (pad00 (t/day now))
      " "
      (pad00 (t/hour now)) ":" (pad00 (t/minute now)) ":" (pad00 (t/second now))
      "." (pad000 (t/milli now)))))

(defn get-space-delimited-values-string
  [values]
  (apply str (interpose " " values)))

(defn get-log-line
  [values]
  (str (get-now-str) ": " (get-space-delimited-values-string values) "\n"))

(process/proc-defn
  log-proc
  [log-line-fn]
  (process/receive!
    msg (do (log-line-fn msg)
          (recur log-line-fn))))

(defn create-log
  [log-fn]
  (let [log (process/spawn log-proc [(comp log-fn get-log-line)])]
    (fn
      [& values]
      (! log values))))

and here's a demo of using it:

(ns otplike-log.core
  (:require [otplike-log.log :as l])
  (:gen-class))

(defn doit
  [this-many-times moniker sleep-ms log]
  (doall (for [n (range this-many-times)]
           (do
             (log moniker n)
             (Thread/sleep sleep-ms)))))

(def doit-slow (partial doit 10 "slow" 1000))

(def doit-fast (partial doit 20 "fast" 500))

(defn log-to-file
  [log-file-spec]
  (fn
    [log-line]
    (spit log-file-spec log-line :append true)))

(defn example-1
  [log]
  (log
    "one"
    (+ 1 2)
    (apply str (interpose "," "three")))
  (doseq [log-message (range 10)]
    (log log-message)))

(defn example-2
  [log]
  (future (doit-fast log))
  (doit-slow log))

(defn -main
  [& args]
  (let [log (l/create-log (log-to-file "log.txt"))]
    (example-1 log)
    (example-2 log)))

Running "-main" produces something like:

2017-09-08 15:18:15.799: one 3 t,h,r,e,e
2017-09-08 15:18:15.801: 0
2017-09-08 15:18:15.801: 1
2017-09-08 15:18:15.802: 2
2017-09-08 15:18:15.803: 3
2017-09-08 15:18:15.803: 4
2017-09-08 15:18:15.804: 5
2017-09-08 15:18:15.804: 6
2017-09-08 15:18:15.805: 7
2017-09-08 15:18:15.806: 8
2017-09-08 15:18:15.806: 9
2017-09-08 15:18:15.807: fast 0
2017-09-08 15:18:15.807: slow 0
2017-09-08 15:18:16.301: fast 1
2017-09-08 15:18:16.801: slow 1
2017-09-08 15:18:16.802: fast 2
2017-09-08 15:18:17.302: fast 3
2017-09-08 15:18:17.801: slow 2
2017-09-08 15:18:17.803: fast 4
2017-09-08 15:18:18.303: fast 5
2017-09-08 15:18:18.802: slow 3
2017-09-08 15:18:18.803: fast 6
2017-09-08 15:18:19.304: fast 7
2017-09-08 15:18:19.803: slow 4
2017-09-08 15:18:19.804: fast 8
2017-09-08 15:18:20.305: fast 9
2017-09-08 15:18:20.803: slow 5
2017-09-08 15:18:20.805: fast 10
2017-09-08 15:18:21.305: fast 11
2017-09-08 15:18:21.804: slow 6
2017-09-08 15:18:21.806: fast 12
2017-09-08 15:18:22.307: fast 13
2017-09-08 15:18:22.804: slow 7
2017-09-08 15:18:22.807: fast 14
2017-09-08 15:18:23.307: fast 15
2017-09-08 15:18:23.805: slow 8
2017-09-08 15:18:23.808: fast 16
2017-09-08 15:18:24.308: fast 17
2017-09-08 15:18:24.805: slow 9
2017-09-08 15:18:24.809: fast 18
2017-09-08 15:18:25.309: fast 19

EuroClojure 2017 Slide 14 Possible Error

On slide 14 the following code is displayed:

(require '[otplike.gen-server :as gs])

; provide initial state (required!)
(defn init []
   [:ok {}])

; handle sync requests
(defn handle-call [message from state]
  (match message 
    [::add a1 a2]
    [:reply (+ a1 a2) state]))

(defn add [server a1 a2]
  (gs/call server [::add a1 a2]))      

(let [[_ server] (gs/start-ns [])]
  (println "2 + 3 = " (add server 2 3)))

In order to get it to compile from a "clean" "lein repl" I had to add the line

(require '[clojure.core.match :refer [match]])

to the top of the code.

When I did, I received the following error:

Exception noproc  otplike.process/self (process.clj:425)

I don't know what this means.

Don't allow parallel receive from process' inbox

It is confusing and error-prone behavior. Maybe we should catch this at runtime. It's worth at least to be mentioned in docs.

(proc-util/execute-proc!!
  (async/go-loop []
    (process/receive!
      _ (do
          (println 1)
          (recur))
      (after 50
             (println "done 1"))))
  (async/go-loop []
    (process/receive!
      _ (do
          (println 2)
          (recur))
      (after 50
             (println "done 2"))))
  (dotimes [_ 30] (! (process/self) :ok))
  (<!! (async/timeout 50)))

Throw special exception when process' fns are called not in process context

It allows setting proper exit reason, especially in cases when process/exit is called with self pid as an argument.

java.lang.NullPointerException during fn call in process/spawn

I wrote simple pmap version for OTPlike with fixed number of workers:

(ns homework.pmap
  (:require
    [otplike.process :as process :refer [!]]
    [otplike.timer :as timer]
    [otplike.proc-util :as proc-util]
    [clojure.core.async :as async :refer [go <!! timeout]]
    [data-miner.util :as util]))


(defn p-map* [f values workers]
  (let [self (process/self)
        call-fn  (fn [idx v]
                   (let [result (try
                                  [idx (f v)]
                                  (catch Throwable e
                                    [idx [:error [:exception {:message (str e)}]]]))]
                     (process/! self result)))]

    (loop [active-workers 0
           last-worker-id 0
           rest-values    values
           results        []]
      (cond

        (= (count results) (count values))
        (->> (sort-by :idx results)
             (map :result))

        (< active-workers workers)
        (let [active-workers (inc active-workers)
              last-worker-id (inc last-worker-id)
              v              (first rest-values)
              rest-values    (rest rest-values)]
          (process/spawn call-fn [last-worker-id v])
          (recur active-workers last-worker-id rest-values results))

        :else
        (process/receive!!

          [idx result]
          (recur (dec active-workers) last-worker-id rest-values (conj results {:idx idx, :result result})))))))


(defn p-map-util [f values workers]
  (let [done-ch (async/chan)]
    (process/spawn
      #(try
         (async/>!! done-ch (p-map* f values workers))
         (finally
          (async/close! done-ch))))
    (async/<!! done-ch)))

I have some questions:

1. When we set workers=3 in this example, we have java.lang.NullPointerException in results:

;;; workers = 3
(p-map-util #(do (<!! (timeout (* % 1000))) :ok) (range 1 (inc 3)) 3)
;;; =>
 (:ok
 [:error [:exception {:message "java.lang.NullPointerException"}]]
 [:error [:exception {:message "java.lang.NullPointerException"}]])

;;; workers= 1
(p-map-util #(do (<!! (timeout (* % 1000))) :ok) (range 1 (inc 3)) 1)
;;; =>
(:ok :ok :ok)

Since the fn does not depend on workers number, it's look like a bug in architecture ?

2. We try to run this p-map* in overload calculation test and the test shows that p-map* uses just only 1 CPU. Are any ideas why it may happens?

Thanks.

simple-supervisor behaviour

Special behaviour for simple_one_for_one strategy.

Add process/request!

As long as otplike has no selective reiceive, API functions using receive! can not be implemented without risk to affect process' inbox.
In such cases process/request! instead of ! + receive! makes it possible to call process and get response without affecting process' inbox, monitoring process and matching on unique request identifier.

supervisor must react on :shutdown exit reason

parent supervisors or out of process code may call (exit :shutdown) on supervisors. it is expected that the supervisor will shut down its children and terminate itself.

currently supervisor ignores such exit reason.

circuit-breaker-like supervision?

I'm thinking of porting some code to otplike but need to keep its behavior as much as possible. Right now I have 3 threads that communicate through 2 buffered streams. Each thread is started by a function called supervise that runs a given thunk in a try-catch and restarts the thunk in case of failure. The restart strategy can be e.g. to wait (min (long (* last-sleep 1.5)) 5-minutes) with a start of 5 seconds, giving an increasing waiting time (5, 7, 10, 15, ... , 300, 300, 300 ...).

I might not want to keep the separation of supervising (i.e. each thread can restart by itself) because supervising the whole with :one-for-all might yield simpler and more robust code. But I need to be able to start the whole thing and keep restarting it in case of failures with similar behavior as described above to ease the load on the system.

I see some knobs on the supervisor, namely :intensity and :period, to control when should the supervisor fail, but this doesn't address my questions.

Is this an uncommon request? Or is this usually handled somehow differently? Maybe I'm expecting the supervisor to do something that it's not supposed to. For the simplest example - if N supervised trees are streaming data from a database and the connection dies (e.g. the database is restarting) and it's down for 2 minutes I don't want all of them to keep restarting in a busy loop, nor do I want the supervisors to fail, just to keep trying until they succeed.

Check if process calling process/link traps exits

If so, :EXIT message must be sent instead of throwing an exception.

Using receive In A Helper Function

Within event_test.clj you'll notice that do-receive is a macro. I had to make it a macro because it did not work as a function. It compiles, but when the tests are executed I get the following error associated with the call to receive!:

java.lang.AssertionError: Assert failed: alts! used not in (go ...) block

The following code is the implementation I tested of do-receive as a function:

(defn do-receive
  ([done-name] (do-receive done-name true 0 false))
  ([done-name unique] (do-receive done-name false unique true))
  ([done-name done-is unique unique-is]
   (op/receive!
     [unique :ok]
     (is unique-is "received [ref :ok] message")

     [:done done-name]
     (is done-is "to-go timeout reached")

     :else
     (is false "unknown message")

     (after 2000 (is false "timeout occurred")))))

The odd thing is that the following similar code executes perfectly in a newly started lein repl:

(require '[otplike.process :as process :refer [! pid->str self]])
(require '[clojure.core.async :as async])

(defn my-receive
  [msg1 msg2 timeout self]
  (process/receive!
    msg (printf "[%s] %s %s%n" self msg1 msg)
    (after timeout
           (printf "[%s] %s %s%n" self msg2 timeout))))

(process/proc-defn
  await-message
  [timeout]
  (my-receive "receive" "timeout" timeout (process/self)))

(defn run []
  (let [pid0 (process/spawn await-message [1000])]
    (println "[main] Started [" pid0 "]")
    (let [pid1 (process/spawn await-message [(async/timeout 1000)])]
      (println "[main] Started [" pid1 "]")
      (let [pid2 (process/spawn await-message [:infinity])]
        (println "[main] Started [" pid2 "]")
        (println "[main] Waiting for 2 seconds")
        (Thread/sleep 2000)
        (println "[main] Sending :stop to [" pid2 "]")
        (! pid2 :stop)
        (println "[main] Waiting for 100mS")
        (Thread/sleep 100)
        (println "[main] Done")))))

(run)

Simple Logging System

I'm trying to implement a simple logging system. The following is the code for the system.

When (-main) is executed only the first log string "one" is logged.

Any suggestions?

(ns otplike-log.core
  (:require [otplike.process :as process :refer [!]])
  (:gen-class))

(defn log-string-to-file
  [log-file-spec]
  (fn
    [log-message]
    (spit log-file-spec (str log-message "\n") :append true)))

(process/proc-defn
  log-proc
  [log-string-fn]
  (process/receive!
    msg (log-string-fn msg)))

(defn create-log
  [log-string-fn]
  (let [log (process/spawn log-proc [log-string-fn])]
    (fn
      [log-string]
      (! log log-string))))

(defn -main
  [& args]
  (let [log (create-log (log-string-to-file "log.txt"))]
    (log "one")
    (log "two")
    (log "three")))

Process spawned with :link-to option isn't linked at the moment when process fn starts execution

The test below fails. pfn1 receives exit message with reason :noproc instead of :abnormal because linking happens after process is started.

(deftest exit-self-reason-is-process-exit-reason
  (let [done (async/chan)
        done1 (async/chan)
        pfn (proc-fn []
              (await-completion done1 150)
              (process/exit (process/self) :abnormal))
        pid (process/spawn pfn [] {})
        pfn1 (proc-fn []
               (async/close! done1)
               (is (= [:exit [pid :abnormal]] (<! (await-message 150)))
                   "process exit reason must be the one passed to exit call")
               (async/close! done))]
    (process/spawn pfn1 [] {:link-to pid :flags {:trap-exit true}})
    (await-completion done 300)))

What happens:

1. pfn waits for pfn1 to start.
2. pfn1 starts.
3. pfn calls process/exit and finishes, so its process doesn't exist anymore.
4. process/spawn (spawned pfn1) fails to link process to pfn's process, sending :noproc to pfn1's process.

Arbitrary process properties (gproc-like registry)

Handling backpressure

Based on some quick reads on the erlang side a message send is fire-and-forget and there's no easy way to provide backpressure on the message queue. What are some other idioms used to handle this scenario:

Process 1 gets data through the network and sends it to process 2.
Process 2 transforms the data and sends it to process 3.
Process 3 sends the data through the network.

If process 1 goes too fast it will eat up a lot of memory in 2 and 3's queues. I have an application where I have N of jobs that consist of 3 threads and 2 queues for the communication between them. The queues are bounded so the threads attempting to put a message in block, thereby creating the necessary backrpressure.

Is the situation same in otplike, messages are fire-and-forget and the queues are always unbounded?

Is there a a "design pattern" used for this? E.g. Process 3 sending messages "ok give me more" to process 1?

map async value

Ability to get a new async-value transforming the result before it is returned from process/await!.
May look like this:

(process/map-async f async-val)

Or as a macro:

(process/with-async [res async-val]
  (inc res))

Tests for gen-server

supervisor behaviour