osnap's People
osnap's Issues
Use snapshot with missing encodings
Error scenario:
Create a scenario using a full encoding specification:
let f = ( +. )
let spec_add = Spec.(float ^> float ^>> Result.float)
let test =
let path = ".osnap/f" in
Test.make ~spec:spec_add ~path ~count:5 ~name:"f" f
let _ =
Osnap.Runner.(run_tests ~encoding:Data_encoding ~mode:Interactive [ test ])
Then remove encoding from the specification:
let spec_add = Spec.(float ^> of_gen QCheck.Gen.float ^>> Result.float)
Error raised:
Uncaught exception:
"Assert_failure src/scenario.ml:76:6"
Raised at Osnap__Scenario.encoding_scenario in file "src/scenario.ml", line 76, characters 6-38
Called from Osnap__Scenario.encoding_scenario in file "src/scenario.ml", line 81, characters 26-48
Called from Osnap__Snapshot.encoding in file "src/snapshot.ml", line 63, characters 35-66
Called from Osnap__Snapshot.decode in file "src/snapshot.ml", line 142, characters 38-53
Called from Osnap__Snapshot.decode_opt in file "src/snapshot.ml", line 146, characters 6-33
Called from Osnap.Runner.run in file "src/osnap.ml", line 214, characters 15-64
Called from Stdlib__list.map in file "list.ml", line 92, characters 20-23
Called from Osnap.Runner.run_tests_with_res in file "src/osnap.ml", line 236, characters 14-52
Called from Osnap.Runner.run_tests in file "src/osnap.ml", line 245, characters 24-66
Interactive mode and `test` stanza
If the executable is listed as a test End_of_file
is raised on Osnap.Runner.take_input
. I don't know if we can force the wait for the input.
Maybe Interactive
should be deleted and provide a binary for this specific mode.
Allow result's extraction of snapshot
For example previous snapshot has result int
add 0 0 0
add 1 1 2
But our new function add
returns the integer and string version:
add 0 0 (0, "0")
add 1 1 (2, "2")
(** [make path spec name f] builds a test
@param count number of application
@param rand random state
@param path where snapshots are stored
@param spec function specification
@param name function name
@param f function to snapshot
*)
val make :
?count:int ->
?rand:Random.State.t ->
path:string ->
spec:('a -> 'b, 'c) Spec.t ->
name:string ->
('a -> 'b) ->
t
Test.make
would take an extra parameter ?extract:('b -> 'c)
.
This would allow the possibility to use snapshot on functions when the type result has changed.
Translate to alcotest tests
Update documentation and restrict visibility through interface files
Improve runner style
The runner is in v0.1 pretty simple. Osnap's runner should take inspiration
from QCheck or Alcotest runner.
Maybe test could also be translated to such runners
Make diff only on results
Diff should be computed only on function results.
Related to #9, parameters will always be equal in prev/next snapshots.
The programmer could introduce a new printer to better debug on errors.
However, printing a parameter should not create a diff.
When the diff will be displayed, function + parameters + result will be printed
with given printers. But internally, errors will only be on results.
New diff should be printed in green
This was forgotten in 0.2.0
Path not found
If the path directory/path given is not found, Osnap raises:
(Sys_error ".osnap/exponentiation: No such file or directory")
.
We should either:
- Give the real absent directory (e.g.
pwd/.osnap/
) - Create the directory? (related to #11)
Refactor tests with common helpers
A lot of code could be refactored, and helpers function could be extracted
for each file.
Better snapshot error management
Provide more 'a spec in module Spec
Provide 'a spec for every 'a supported in QCheck.
What happens when switching from Marshal/Data_encoding snapshots
Error scenario:
Create a snapshot with Data_encoding:
open Osnap
let f = ( +. )
let spec_add = Spec.(float ^> float ^>> Result.float)
let test =
let path = ".osnap/f" in
Test.make ~spec:spec_add ~path ~count:5 ~name:"f" f
let _ = Osnap.Runner.(run_tests ~encoding:Data_encoding ~mode:Interactive [ test ])
Then change encoding for Marshal:
let _ = Osnap.Runner.(run_tests ~encoding:Marshal ~mode:Interactive [ test ])
The snapshot in generated twice, like it did not exist on disk. Same issue applies when switching Marshal - Data_encoding.
Specific module for encoding and decoding
Encoding and decoding are split between Osnap
and Memory
. They should be merge into a specific module in order to:
- Facilitate error management on I/O
- Prepare the field for encoding with Data-encoding
Allows optional path
A default path should be considered for snapshots.
Provide Result default types
Currently:
let add x y = x + y
let spec_add =
Spec.(int ^> int ^>> build_result string_of_int)
We would prefer:
let add x y = x + y
let spec_add =
Spec.(int ^> int ^>> Res.int)
Interactive mode should display diff instead of new version
No support for labelled parameters
I don't think we can reproduce the following behaviors with labelled parameters:
let rec spec_to_scenario :
type fn r. rand:Random.State.t -> (fn, r) Spec.t -> fn -> (fn, r) t =
fun ~rand spec f ->
match spec with
| Arrow ({ gen; _ }, Result _) -> (
let x = Gen.generate1 ~rand gen in
try
let f = f x in
Cons (x, Res (Ok f))
with e -> Cons (x, Res (Error (Printexc.to_string e))))
| Arrow ({ gen; _ }, spec) ->
let x = Gen.generate1 ~rand gen in
let f = f x in
Cons (x, spec_to_scenario ~rand spec f)
| Result _ -> Res (Ok f)
The only solution we currently have is to write:
let f ~x ~y = x + y
let test =
let open Osnap in
let small_int =
Spec.
{
gen = QCheck.Gen.small_int;
printer = Some string_of_int;
encoding = None;
}
in
let spec = Spec.(small_int ^> small_int ^>> Result.int) in
let path = ".osnap/exponentiation" in
Test.make ~spec ~path ~count:5 ~name:"exponentiation" (fun x y -> f ~x ~y)
Support exceptions raised by functions tested
We could have the exception as the scenario's result. However, this is ill-typed exn <> r
.
-
Solution 1:
Maybe the solution could be to generate a scenario again and again when the function raises an exception. But, this would need to be limited by a timeout (how could we determine that timeout?), and could we generate empty scenarios if the function raises too much exceptions? -
Solution 2:
type 'a res = Ok of 'a | Error of exn
?
module Interpret is useless
The only thing usefull is type args and spec_to_args.
Type expr is useless and interpretation can be done without the translation from args to expr.
Readme with small example
Keep generated values improving the code's coverage
@clarus suggested this idea:
We aim for a code coverage (~80 to 100% I guess), and we only keep generated values that improves the code's coverage.
Better CI
The CI should:
- do an installation from scratch
- build the project
- run the test
- display the coverage on the README ?
Encoding integers too big
Integers generators generates values respecting the range of Data_encoding.int31
.
However, the function's result could be outside this range and raise:
(Invalid_argument "Json_encoding.construct: int out of range")
The issue could be the same with a lot of encodings, should we catch errors from Data_encoding
or leave it as is?
Remove patdiff dependency
It bring too much dependencies in the project with the JS's stuffs.
Allows no printer for parameters
(** ['a spec] combines an ['a gen] and a printer *)
type 'a spec = { gen : 'a gen; printer : 'a printer option}
(** [t] is the specification type, describing a function.
Thus [t] declaration must end with {! (^>>) }. *)
type ('fn, 'r) t =
| Result : 'a printer -> ('a, 'a) t
| Arrow : 'a spec * ('fn, 'r) t -> ('a -> 'fn, 'r) t
Parameters should not be printed. Since new snapshots will use previous
parameters, we know that they are equals, it's not mandatory to print them.
However, the result printer must remains.
Create specific module Runner
Split code between
- src
---- core
---- runner
Allowing to separate the code between core utilities and user interface. Folder Runner could also contains several runners.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.