vch9 / osnap Goto Github PK

Error scenario:
Create a scenario using a full encoding specification:

let f = ( +. )

let spec_add = Spec.(float ^> float ^>> Result.float)

let test =
  let path = ".osnap/f" in
  Test.make ~spec:spec_add ~path ~count:5 ~name:"f" f

let _ =
  Osnap.Runner.(run_tests ~encoding:Data_encoding ~mode:Interactive [ test ])

Then remove encoding from the specification:

let spec_add = Spec.(float ^> of_gen QCheck.Gen.float ^>> Result.float)

Error raised:

Uncaught exception:
  
  "Assert_failure src/scenario.ml:76:6"

Raised at Osnap__Scenario.encoding_scenario in file "src/scenario.ml", line 76, characters 6-38
Called from Osnap__Scenario.encoding_scenario in file "src/scenario.ml", line 81, characters 26-48
Called from Osnap__Snapshot.encoding in file "src/snapshot.ml", line 63, characters 35-66
Called from Osnap__Snapshot.decode in file "src/snapshot.ml", line 142, characters 38-53
Called from Osnap__Snapshot.decode_opt in file "src/snapshot.ml", line 146, characters 6-33
Called from Osnap.Runner.run in file "src/osnap.ml", line 214, characters 15-64
Called from Stdlib__list.map in file "list.ml", line 92, characters 20-23
Called from Osnap.Runner.run_tests_with_res in file "src/osnap.ml", line 236, characters 14-52
Called from Osnap.Runner.run_tests in file "src/osnap.ml", line 245, characters 24-66

If the executable is listed as a test End_of_file is raised on Osnap.Runner.take_input. I don't know if we can force the wait for the input.

Maybe Interactive should be deleted and provide a binary for this specific mode.

For example previous snapshot has result int

add 0 0 0
add 1 1 2

But our new function add returns the integer and string version:

add 0 0 (0, "0")
add 1 1 (2, "2")

(** [make path spec name f] builds a test
      @param count number of application
      @param rand random state
      @param path where snapshots are stored 
      @param spec function specification
      @param name function name
      @param f function to snapshot
  *)
  val make :
    ?count:int ->
    ?rand:Random.State.t ->
    path:string ->
    spec:('a -> 'b, 'c) Spec.t ->
    name:string ->
    ('a -> 'b) ->
    t

Test.make would take an extra parameter ?extract:('b -> 'c).

This would allow the possibility to use snapshot on functions when the type result has changed.

The runner is in v0.1 pretty simple. Osnap's runner should take inspiration
from QCheck or Alcotest runner.

Maybe test could also be translated to such runners

Diff should be computed only on function results.
Related to #9, parameters will always be equal in prev/next snapshots.

The programmer could introduce a new printer to better debug on errors.
However, printing a parameter should not create a diff.

When the diff will be displayed, function + parameters + result will be printed
with given printers. But internally, errors will only be on results.

This was forgotten in 0.2.0

If the path directory/path given is not found, Osnap raises:
(Sys_error ".osnap/exponentiation: No such file or directory").

We should either:

• Give the real absent directory (e.g. pwd/.osnap/)
• Create the directory? (related to #11)

A lot of code could be refactored, and helpers function could be extracted
for each file.

What would happens:

• The Spec.t does not correspond to the values in the Snapshot
• The snapshot is corrupted
• Reading/writing errors

Sub-issues: #24 #23 #21

Provide 'a spec for every 'a supported in QCheck.

Error scenario:

Create a snapshot with Data_encoding:

open Osnap

let f = ( +. )

let spec_add = Spec.(float ^> float ^>> Result.float)

let test =
  let path = ".osnap/f" in
  Test.make ~spec:spec_add ~path ~count:5 ~name:"f" f

let _ = Osnap.Runner.(run_tests ~encoding:Data_encoding ~mode:Interactive [ test ])

Then change encoding for Marshal:

let _ = Osnap.Runner.(run_tests ~encoding:Marshal ~mode:Interactive [ test ])

The snapshot in generated twice, like it did not exist on disk. Same issue applies when switching Marshal - Data_encoding.

Encoding and decoding are split between Osnap and Memory. They should be merge into a specific module in order to:

• Facilitate error management on I/O
• Prepare the field for encoding with Data-encoding

A default path should be considered for snapshots.

Currently:

let add x y = x + y

let spec_add =
  Spec.(int ^> int ^>> build_result string_of_int)

We would prefer:

let add x y = x + y

let spec_add =
  Spec.(int ^> int ^>> Res.int)

I don't think we can reproduce the following behaviors with labelled parameters:

let rec spec_to_scenario :
    type fn r. rand:Random.State.t -> (fn, r) Spec.t -> fn -> (fn, r) t =
 fun ~rand spec f ->
  match spec with
  | Arrow ({ gen; _ }, Result _) -> (
      let x = Gen.generate1 ~rand gen in
      try
        let f = f x in
        Cons (x, Res (Ok f))
      with e -> Cons (x, Res (Error (Printexc.to_string e))))
  | Arrow ({ gen; _ }, spec) ->
      let x = Gen.generate1 ~rand gen in
      let f = f x in
      Cons (x, spec_to_scenario ~rand spec f)
  | Result _ -> Res (Ok f)

The only solution we currently have is to write:

let f ~x ~y = x + y

let test =
  let open Osnap in
  let small_int =
    Spec.
      {
        gen = QCheck.Gen.small_int;
        printer = Some string_of_int;
        encoding = None;
      }
  in
  let spec = Spec.(small_int ^> small_int ^>> Result.int) in
  let path = ".osnap/exponentiation" in

  Test.make ~spec ~path ~count:5 ~name:"exponentiation" (fun x y -> f ~x ~y)

We could have the exception as the scenario's result. However, this is ill-typed exn <> r.

• Solution 1:
  Maybe the solution could be to generate a scenario again and again when the function raises an exception. But, this would need to be limited by a timeout (how could we determine that timeout?), and could we generate empty scenarios if the function raises too much exceptions?

• Solution 2:

type 'a res = Ok of 'a | Error of exn

?

The only thing usefull is type args and spec_to_args.
Type expr is useless and interpretation can be done without the translation from args to expr.

@clarus suggested this idea:

We aim for a code coverage (~80 to 100% I guess), and we only keep generated values that improves the code's coverage.

The CI should:

• do an installation from scratch
• build the project
• run the test
• display the coverage on the README ?

Integers generators generates values respecting the range of Data_encoding.int31.

However, the function's result could be outside this range and raise:
(Invalid_argument "Json_encoding.construct: int out of range")

The issue could be the same with a lot of encodings, should we catch errors from Data_encoding or leave it as is?

It bring too much dependencies in the project with the JS's stuffs.

(** ['a spec] combines an ['a gen] and a printer *)
type 'a spec = { gen : 'a gen; printer : 'a printer option}

(** [t] is the specification type, describing a function.
    Thus [t] declaration must end with {! (^>>) }. *)
type ('fn, 'r) t =
  | Result : 'a printer -> ('a, 'a) t
  | Arrow : 'a spec * ('fn, 'r) t -> ('a -> 'fn, 'r) t

Parameters should not be printed. Since new snapshots will use previous
parameters, we know that they are equals, it's not mandatory to print them.
However, the result printer must remains.

Split code between

- src
---- core
---- runner

Allowing to separate the code between core utilities and user interface. Folder Runner could also contains several runners.