dnlserrano / exavier Goto Github PK

View Code? Open in Web Editor NEW

96.0 96.0 6.0 290 KB

Elixir mutation testing library

License: MIT License

Elixir 100.00%

exavier's People

Contributors

Stargazers

Watchers

Forkers

kingofrostov exavier-elixir ordepdev iaguirre88 filipevarjao cantido

exavier's Issues

No results/GenServer terminating (killed)

Hi! Whenever I run MIX_ENV=test mix exavier.test, this is the output:

..21:47:16.995 [error] GenServer #PID<0.811.0> terminating
** (stop) killed
Last message: {:EXIT, #PID<0.667.0>, :killed}
21:47:16.995 [error] GenServer #PID<0.827.0> terminating
** (stop) killed
Last message: {:EXIT, #PID<0.664.0>, :killed}
21:47:16.995 [error] GenServer Exavier.Server terminating
** (stop) exited in: Task.Supervised.stream(5000)
    ** (EXIT) time out
    (elixir 1.11.3) lib/task/supervised.ex:304: Task.Supervised.stream_reduce/7
    (elixir 1.11.3) lib/enum.ex:3473: Enum.reverse/1
    (elixir 1.11.3) lib/enum.ex:3066: Enum.to_list/1
    (exavier 0.3.0) lib/exavier/server.ex:59: Exavier.Server.handle_call/3
    (stdlib 3.14.2) gen_server.erl:715: :gen_server.try_handle_call/4
    (stdlib 3.14.2) gen_server.erl:744: :gen_server.handle_msg/6
    (stdlib 3.14.2) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.95.0>): :xmen
** (EXIT from #PID<0.95.0>) exited in: Task.Supervised.stream(5000)
    ** (EXIT) time out

Can this be linked to #1?

This is my Erlang/Elixir info:

$ elixir --version
Erlang/OTP 23 [erts-11.2.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

Elixir 1.11.3 (compiled with Erlang/OTP 23)

Thank you!

Change configuration to avoid using application configuration

Howdy!

So I decided to give this a quick try today, and it turns out that having to set configuration in a config.exs file is a bit of an issue since the code I'm trying to test is a library, and I don't want this config ending up in other people's applications. It would be good if this library configuration could follow the configuration guidelines set out in the documentation here so this problem could be avoided.

For example, I'm trying to test benchee, but we don't even have any mix config in there (for the reasons laid out by the documentation there), and it won't run at the moment without some configuration (custom mappings), so this is a big of a sticky issue for users initial experience. I think either a dotfile that can be .gitignore'd (like Credo does) or some sort of function call somewhere would be a nice place for this configuration to reside.

Thanks for the great start towards a really nice mutation testing library for our community!

Could not find module inferred from test file

Exavier does not infer correctly some modules with dots in their names.

The error message:

MIX_ENV=test mix exavier.test

13:50:30.016 [error] Could not find module Elixir.ExavierCheck.Main inferred from test file test/exavier_check/main_test.exs. You can define your overrides using the :test_files_to_modules option in exavier.

13:50:30.019 [error] GenServer Exavier.Server terminating
** (MatchError) no match of right hand side value: :ok
    (exavier 0.3.0) lib/exavier/cover.ex:8: Exavier.Cover.lines_to_mutate/2
    (exavier 0.3.0) lib/exavier/server.ex:20: anonymous fn/2 in Exavier.Server.handle_call/3
    (elixir 1.13.0) lib/enum.ex:2396: Enum."-reduce/3-lists^foldl/2-0-"/3
    (exavier 0.3.0) lib/exavier/server.ex:18: Exavier.Server.handle_call/3
    (stdlib 3.16.1) gen_server.erl:721: :gen_server.try_handle_call/4
    (stdlib 3.16.1) gen_server.erl:750: :gen_server.handle_msg/6
    (stdlib 3.16.1) proc_lib.erl:226: :proc_lib.init_p_do_apply/3

I prepared a repository to reproduce the issue: https://github.com/denispeplin/exavier_check

To reproduce, checkout the repository and run

mix deps.get
MIX_ENV=test mix exavier.test

Test for ROR1 is testing AOR1 instead

On this line:

exavier/test/exavier/mutators/ror1_test.exs

Line 4 in 4fb6796

@subject Exavier.Mutators.AOR1

The subject should be ROR1, not AOR1. The test cases are also testing AOR1's replacements.

I'm in the middle of another PR; I wanted to write this down before I forgot.

Argument error when reporting results

Hey @dnlserrano! I tried this out on my norm library and I got an argument error from the reporter. It looks like its trying to convert a string to an existing atom and it can't find the atom or something similar. Here's the full output:

keathley in ~/D/norm on improve-docs
λ env MIX_ENV=test mix exavier.test
Compiling 11 files (.ex)
Generated norm app

08:47:04.990 [error] GenServer :exavier_reporter terminating
** (ArgumentError) argument error
    :erlang.binary_to_existing_atom("Elixir.Norm.Selection", :utf8)
    (exavier) lib/exavier.ex:75: Exavier.string_to_elixir_module/1
    (exavier) lib/exavier/reporter.ex:34: Exavier.Reporter.handle_cast/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_cast", {:test_finished, %ExUnit.Test{case: Norm.SelectionTest, logs: "", module: Norm.SelectionTest, name: :"test generation can generate values", state: nil, tags: %{async: true, case: Norm.SelectionTest, describe: "generation", describe_line: 58, file: "/Users/keathley/Development/norm/test/norm/selection_test.exs", line: 59, module: Norm.SelectionTest, registered: %{}, test: :"test generation can generate values", test_type: :test}, time: 190}}}
State: %Exavier.Reporter{all_failures: [], mutated_modules: %{}}
** (EXIT from #PID<0.92.0>) an exception was raised:
    ** (ArgumentError) argument error
        :erlang.binary_to_existing_atom("Elixir.Norm.Selection", :utf8)
        (exavier) lib/exavier.ex:75: Exavier.string_to_elixir_module/1
        (exavier) lib/exavier/reporter.ex:34: Exavier.Reporter.handle_cast/2
        (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
        (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
        (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

I'm running on elixir 1.9.1 and OTP 22 if that helps. I've also tested this with the most recent version of exavier on hex and with the master branch. The failure is non-deterministic and I suspect that just reporting whatever test it receives first since they're running async. I suspect that it would be safe to use to_atom here since this will really only be run in test mode and we don't really have to worry about DOSing the atom table. But I figured I'd leave that up to you.

cannot add sync case to test suite after the suite starts running

When I try running mix exavier.test, it terminates with an error like this (with a different module named each time):

[error] Task #PID<0.6433.0> started from Exavier.Server terminating
** (RuntimeError) cannot add sync case named MyApp.SomeTest to test suite after the suite starts running
    (ex_unit 1.12.1) lib/ex_unit/server.ex:21: ExUnit.Server.add/2
    (elixir 1.12.1) src/elixir_module.erl:369: anonymous fn/5 in :elixir_module.expand_callback/6
    (elixir 1.12.1) src/elixir_module.erl:368: :elixir_module.expand_callback/6
    (stdlib 3.14.2.1) lists.erl:1267: :lists.foldl/3
    (elixir 1.12.1) src/elixir_module.erl:150: :elixir_module.compile/5
    (stdlib 3.14.2.1) erl_eval.erl:680: :erl_eval.do_apply/6
    (elixir 1.12.1) src/elixir.erl:280: :elixir.recur_eval/3
    (elixir 1.12.1) src/elixir.erl:265: :elixir.eval_forms/3
    (elixir 1.12.1) src/elixir_compiler.erl:46: :elixir_compiler.eval_forms/3
    (elixir 1.12.1) src/elixir_lexical.erl:15: :elixir_lexical.run/3
    (elixir 1.12.1) src/elixir_compiler.erl:18: :elixir_compiler.quoted/3
    (elixir 1.12.1) lib/code.ex:1261: Code.require_file/2
    (exavier 0.3.0) lib/exavier/server.ex:49: anonymous fn/5 in Exavier.Server.handle_call/3
    (elixir 1.12.1) lib/enum.ex:930: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir 1.12.1) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
    (elixir 1.12.1) lib/task/supervised.ex:35: Task.Supervised.reply/5
    (stdlib 3.14.2.1) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Function: &:erlang.apply/2
    Args: [#Function<2.82788263/1 in Exavier.Server.handle_call/3>, [{"test/my_app/some_test.exs", %{file: '/Users/whoever/my_app/apps/my_app/lib/my_app/some.ex', lines_to_mutate: [16, 22, 25, 26], module: MyApp.Some}}]]

Exavier 0.3.0
Elixir 1.12.1
Erlang/OTP 23

Handle negative numbers properly

Hey @dnlserrano, thanks for your work on exavier! Our team'd like to give it a shot.

We noticed that in case when the function returns negative number, e.g:

def something(_arg1), do: -1

exavier will mutate this -1 into *1 and here's gonna be an error like so:

** (CompileError): undefined function */1

So we fixed it in our fork in this PR.

Skip test files

Is there a way to skip certain test files altogether? Not all of my test files map one-to-one to a module. For example, I have some test files that only test protocol implementations in my project, so there is no Elixir module to be found. I tried setting the values in the :test_files_to_modules option to nil or false, but that doesn't seem to be supported.

Don't mutate and test all in one go

More context on this original post over at ElixirForum.com where I first explained it.

Right now I’m not running each test ... do individually but instead I’m running the whole test module (e.g., HelloWorldTest). This has one clear disadvantage, which I’ll explain below with an example:

defmodule HelloWorld do
  def sum(a, b) do: a + b
  def divide(a, b), do: div(a, b)
end

defmodule HelloWorldTest do
  test "when testing sum" do
    assert HelloWorld.sum(3, 0) == 3
  end

  test "when testing divide" do
    assert HelloWorld.divide(5, 2) == 3
  end
end

If I change code to the following:

defmodule HelloWorld do
  def sum(a, b) do: a - b # changed from + to - via AOR1
  def divide(a, b), do: div(a, b)
end

I will be running the tests for both tests, instead of just running the test for sum/2 (i.e., "when testing sum", which was the only one for which the corresponding source code changed). In order to try and maximise the amount of mutations I can catch with running the entire test module, I mutate all in one go. Does that make sense? Maybe it doesn’t… 🤦‍♂️ AFAIU, finding out what tests I should run per source code change is hard (or at least not trivial? 😅). But I might not be seeing something very obvious.

Hope people can help out with ideas for this one. Maybe traversing tests and annotating to keep track of what operators are present in each and hence are influenced/swing based on mutations performed? 🤔

Handle function references properly

Hi!

Thanks for your work on exavier, it looks like a very interesting project! :)

I'm aware that this is mostly still a proof-of-concept, but when I just tried to run it on one of our projects, it failed because it generated invalid code. Specifically, it interpreted the / in a function reference (i.e. something like &some_function/1) as a division symbol and tried to mutate it to &some_function * 1, which is not valid elixir code anymore...

So I guess, I would expect function references to be left alone, or alternatively maybe compilation errors could be caught and respective mutations be discarded?

dnlserrano / exavier Goto Github PK

exavier's People

Contributors

Stargazers

Watchers

Forkers

exavier's Issues

No results/GenServer terminating (killed)

Change configuration to avoid using application configuration

Could not find module inferred from test file

Test for ROR1 is testing AOR1 instead

Argument error when reporting results

cannot add sync case to test suite after the suite starts running

Handle negative numbers properly

Skip test files

Don't mutate and test all in one go

Handle function references properly

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

Jobs