ligurio / elle-cli Goto Github PK

View Code? Open in Web Editor NEW

25.0 25.0 5.0 2.76 MB

command-line frontend to transactional consistency checkers for black-box databases

License: Other

Clojure 83.21% Shell 16.79%

clojure elle jepsen knossos

elle-cli's People

Contributors

Stargazers

Watchers

Forkers

gerrrr mprimi sqaunderhood hciniramy

elle-cli's Issues

Failed to process history for checker elle-list-append

sergeyb@pony:~/sources/ljepsen/elle-cli$ lein run --model elle-list-append histories/elle/paper-example.json
Compiling elle_cli.cli                                              
                                                                    
java.lang.IllegalArgumentException: No matching clause: append
        at elle.list_append$op_internal_case$fn__1816.invoke(list_append.clj:134)
        at clojure.lang.PersistentVector.reduce(PersistentVector.java:343)
        at clojure.core$reduce.invokeStatic(core.clj:6829)
        at clojure.core$reduce.invoke(core.clj:6812)     
        at elle.list_append$op_internal_case.invokeStatic(list_append.clj:133)
        at elle.list_append$op_internal_case.invoke(list_append.clj:122)
        at clojure.core$keep$fn__8582.invoke(core.clj:7339)   
        at clojure.lang.LazySeq.sval(LazySeq.java:42)      
        at clojure.lang.LazySeq.seq(LazySeq.java:51) 
        at clojure.lang.RT.seq(RT.java:535)

Print unknown status as a string not Clojure keyword

Status :unknown means checker was unable to complete the analysis; e.g. it ran out of memory.
Now elle-cli prints it as a Clojure keyword (with colon), it would be better to print it without colon.

suggestion: a gen command

Is it possible to create a gen command that generates a set of transaction history in json? The elle doc mentioned about this function but I'm not sure how to use it (I know lisp but I don't know anything about closure). It would be a great option to generate the sequence and feed it into unit test for db development.

Fix jepsen-long-fork model

jepsen-long-fork model is mentioned in documentation and usage, but actually is broken.

elle-cli/src/elle_cli/cli.clj

Lines 174 to 178 in d23bf96

 "jepsen" (case model-name 

 "jepsen-bank" (jepsen-model/check-safe (checker-fn {:negative-balances? true}) nil history) 

 "jepsen-set-full" (jepsen-model/check-safe (checker-fn) nil history) 

 "jepsen-counter" (jepsen-model/check-safe (checker-fn) nil history) 

 ))))

Fix jepsen-set model

jepsen-set model is mentioned in documentation and usage, but actually is broken.

elle-cli/src/elle_cli/cli.clj

Lines 174 to 178 in d23bf96

 "jepsen" (case model-name 

 "jepsen-bank" (jepsen-model/check-safe (checker-fn {:negative-balances? true}) nil history) 

 "jepsen-set-full" (jepsen-model/check-safe (checker-fn) nil history) 

 "jepsen-counter" (jepsen-model/check-safe (checker-fn) nil history) 

 ))))

`elle-rw-registers` always returns true

Example trace that should fail, but passes

P1 and P2 read nil
P1 writes 100
P2 writes 200
P1 reads 200
P2 reads 100

I hope we agree this case should fail under most consistency models.

{:type :invoke, :f :txn :value [[:r :x nil]],   :process 1, :index 1}
{:type :ok,     :f :txn :value [[:r :x nil]],   :process 1, :index 2}
{:type :invoke, :f :txn :value [[:r :x nil]],   :process 2, :index 1}
{:type :ok,     :f :txn :value [[:r :x nil]],   :process 2, :index 2}
{:type :invoke, :f :txn :value [[:w :x 100]],   :process 1, :index 3}
{:type :ok,     :f :txn :value [[:w :x 100]],   :process 1, :index 4}
{:type :invoke, :f :txn :value [[:w :x 200]],   :process 2, :index 3}
{:type :ok,     :f :txn :value [[:w :x 200]],   :process 2, :index 4}
{:type :invoke, :f :txn :value [[:r :x 100]],   :process 1, :index 5}
{:type :ok,     :f :txn :value [[:r :x 200]],   :process 1, :index 6}
{:type :invoke, :f :txn :value [[:r :x 200]],   :process 2, :index 5}
{:type :ok,     :f :txn :value [[:r :x 100]],   :process 2, :index 6}

for mod in strict-serializable consistent-view read-committed serializable snapshot-isolation; do  
  java -jar elle-cli-0.1.2-standalone.jar -m elle-rw-register --cycle-search-timeout 5000 -c ${mod} -v  rw-register.edn; 
done

{"valid?":true}
{"valid?":true}
{"valid?":true}
{"valid?":true}
{"valid?":true}

Tested with 0.1.3 and 0.1.2, same outcome.

Remove checker name in options with models name

There is no sense to include a checker name to a model name. Checker is specified in a model description in a documentation.

Add checker of sequential history

Implementations:

Native binaries with GraalVM

Hi, have you considered using GraalVM to avoid the slow start up from and the dependency on JVM?

Describe operation generators for models in a README

Default to headless mode

elle-cli should probably run by default in (Java) headless mode, since it's a command-line application without a GUI.

Currently, it does not and this creates a couple of inconveniences:

MacOS: when Elle-cli is running there is an application (e.g. shows in alt-tab list of running applications). And it steals focus from the currently active application. So if you have it running in background on a bunch of traces, it's impossible to do anything else, because it keeps going foreground.

Linux (Ubuntu 20.04 with java-default-headless installed):

Caused by: java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
	at java.desktop/sun.awt.HeadlessToolkit.getMenuShortcutKeyMask(HeadlessToolkit.java:135)
	at rhizome.viz__init.load(Unknown Source)
	at rhizome.viz__init.<clinit>(Unknown Source)
	... 126 more

There is a simple workaround that fixes both cases:

java -Djava.awt.headless=true -jar ${ELLE_CLI_JAR}

But it would be nice if Elle-cli was headless-ready out of the box.

More info: https://www.baeldung.com/java-headless-mode

Remove model jepsen-set OR jepsen-set-full

jepsen-set-full and jepsen-set uses the same history format, see - set-full
and set.

Need to understand a difference and probably leave a single one.

Add checker of comments history

See https://github.com/jepsen-io/jepsen/blob/main/cockroachdb/src/jepsen/cockroach/comments.clj

Examples of history:

Remove model knossos-register

There are some options for the same or similar models:

elle-rw-register and knossos-register uses the same history format, as far as I understand elle-rw-register is more rigorous and has a rough formalization of its soundness proof written in Isabelle/HOL, see a source code and proof.

Pass multiple consistency models and anomalies in command line to elle's checkers

not caught exception on running Elle with list-append history

sergeyb@pony:~/sources/ljepsen/elle-cli$ java -jar target/elle-cli-0.1.0-standalone.jar --model elle-list-append histories/elle/paper-example.edn                                                           
INFO [2021-12-28 20:59:39,428] main - elle.txn Timing out search for :G1c-realtime in SCC of 3 transactions                             
                                                                    
java.lang.IllegalStateException: Explainer

Trouble in processing some Elle list-append histories

Hi,

While playing with elle-cli, I hit the following error with some histories:

java.lang.IllegalStateException: Explainer
...
was unable to explain the relationship between {:process 1, :type :ok, :f nil, :value [[:append 3 1] [:append 2 4] [:r 4 []]], :index 6} and #knossos.op.Op{:process 0, :type :ok, :f nil, :value [[:r 2 []] [:r 3 []] [:append 4 0]], :index 8}
	at elle.core$explain_cycle_pair_data.invokeStatic(core.clj:402)
	at elle.core$explain_cycle_pair_data.invoke(core.clj:396)
...

One example of such a history is:

{:index 1, :type :invoke, :process 2, :value [[ :append 4 2] [ :append 5 5] ]}
{:index 2, :type :invoke, :process 0, :value [[ :r 2 nil] [ :r 3 nil] [ :append 4 0] ]}
{:index 3, :type :invoke, :process 1, :value [[ :append 3 1] [ :append 2 4] [ :r 4 nil] ]}
{:index 4, :type :ok, :process 2, :value [[ :append 4 2] [ :append 5 5] ]}
{:index 5, :type :invoke, :process 2, :value [[ :append 2 8] ]}
{:index 6, :type :ok, :process 1, :value [[ :append 3 1] [ :append 2 4] [ :r 4 [ ]] ]}
{:index 7, :type :invoke, :process 1, :value [[ :append 3 7] [ :append 4 10] ]}
{:index 8, :type :ok, :process 0, :value [[ :r 2 [ ]] [ :r 3 [ ]] [ :append 4 0] ]}
{:index 9, :type :ok, :process 2, :value [[ :append 2 8] ]}
{:index 10, :type :ok, :process 1, :value [[ :append 3 7] [ :append 4 10] ]}

I tried the same directly with Elle and the processing went fine, so I believe that this is something to do how elle-cli imports history files.

After copy pasting the code to read history from Elle tests https://github.com/jepsen-io/elle/blob/main/test/elle/core_test.clj#L14, the problem went away.

diff --git a/src/elle_cli/cli.clj b/src/elle_cli/cli.clj
index d43cdb2..dc479b4 100644
--- a/src/elle_cli/cli.clj
+++ b/src/elle_cli/cli.clj
@@ -27,17 +27,10 @@
   expecting it to be a collection of op maps. If the initial character is {,
   loads it piecewise as op maps."
   [filepath]
-  (let [h (with-open [r (PushbackReader. (io/reader filepath))]
-            (->> (repeatedly #(edn/read {:eof nil} r))
-                 (take-while identity)
-                 vec))
-        ; Handle the presence or absence of an enclosing vector.
-        h (if (and (= 1 (count h))
-                   (sequential? (first h)))
-            (vec (first h))
-            h)]
-    ; Normalize ops.
-    (history/parse-ops h)))
+    (with-open [r (PushbackReader. (io/reader filepath))]
+    (->> (repeatedly #(edn/read {:eof nil} r))
+         (take-while identity)
+         vec)))
 
 (defn vl
   [v]

Being new to both Elle and Clojure I can't say for sure if the above change is correct for all models, but it seems to work
for elle-list-append.

To reproduce:

Paste the above history in test.edn
Run lein run --model elle-list-append --format edn --consistency-models repeatable-read test.edn

Plots not created (`elle-list-append`)

Plot (png or SVG) are not produced as a result of a trace with consistency violations.

Example:

{:index 1 :type :invoke, :value [[:append x 8] [:r x []]]}
{:index 2 :type :ok,     :value [[:append x 8] [:r x [8]]]}
{:index 3 :type :invoke, :value [[:r x nil]]}
{:index 4 :type :ok,     :value [[:r x [1 8]]]}

java -jar elle-cli-0.1.3-standalone.jar -m elle-list-append --cycle-search-timeout 5000  -v -p svg -d . list.edn
{"valid?":false,
 "anomaly-types":["incompatible-order"],
 "anomalies":
 {"incompatible-order":[{"key":"x", "values":[[8], [1, 8]]}]},
 "not":["read-atomic", "read-committed"],
 "also-not":
 ["ROLA", "causal-cerone", "consistent-view", "cursor-stability",
  "forward-consistent-view", "monotonic-atomic-view",
  "monotonic-snapshot-read", "monotonic-view",
  "parallel-snapshot-isolation", "prefix", "repeatable-read",
  "serializable", "snapshot-isolation", "strict-serializable",
  "strong-session-serializable", "strong-session-snapshot-isolation",
  "strong-snapshot-isolation", "update-serializable"]}

No graph is created (same result with -p png).

The only output file (not mentioned in documentation) is ./incompatible-order/x.html

Exit with non-zero exit code if any history fails verification

Elle-cli currently exits with code 0 if any analysis is performed.

This makes it a little hard to script around it.
e.g., one has to parse the output and look for the '{"valid?":true}' string.

Suggestion: exit with a non-zero exit code if any of the analyses was invalid.

Fix publishing workflow

make sure it really works
don't put run-elle-cli to zip archive

	"jepsen" (case model-name
	"jepsen-bank" (jepsen-model/check-safe (checker-fn {:negative-balances? true}) nil history)
	"jepsen-set-full" (jepsen-model/check-safe (checker-fn) nil history)
	"jepsen-counter" (jepsen-model/check-safe (checker-fn) nil history)
	))))

ligurio / elle-cli Goto Github PK

elle-cli's People

Contributors

Stargazers

Watchers

Forkers

elle-cli's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs