GithubHelp home page GithubHelp logo

nette / tester Goto Github PK

View Code? Open in Web Editor NEW
446.0 40.0 71.0 1.44 MB

Tester: enjoyable unit testing in PHP with code coverage reporter. 🍏🍏🍎🍏

Home Page: https://tester.nette.org

License: Other

PHP 95.32% HTML 4.68%
nette nette-framework tester unit-testing phpunit php assertions code-coverage xdebug phpdbg pcov

tester's Introduction

Nette Tester: enjoyable unit testing

Downloads this Month Tests Latest Stable Version License

Introduction

Nette Tester is a productive and enjoyable unit testing framework. It's used by the Nette Framework and is capable of testing any PHP code.

Documentation is available on the Nette Tester website. Read the blog for new information.

Do you like Nette Tester? Are you looking forward to the new features?

Buy me a coffee

Thank you!

Installation

The recommended way to install Nette Tester is through Composer:

composer require nette/tester --dev

Alternatively, you can download the tester.phar file.

  • Nette Tester 2.5 is compatible with PHP 8.0 to 8.3
  • Nette Tester 2.4 is compatible with PHP 7.2 to 8.2
  • Nette Tester 2.3 is compatible with PHP 7.1 to 8.0
  • Nette Tester 2.1 & 2.2 is compatible with PHP 7.1 to 7.3
  • Nette Tester 2.0 is compatible with PHP 5.6 to 7.3

Collecting and processing code coverage information depends on Xdebug or PCOV extension, or PHPDBG SAPI.

Writing Tests

Imagine that we are testing this simple class:

class Greeting
{
	function say($name)
	{
		if (!$name) {
			throw new InvalidArgumentException('Invalid name.');
		}
		return "Hello $name";
	}
}

So we create test file named greeting.test.phpt:

require 'src/bootstrap.php';

use Tester\Assert;

$h = new Greeting;

// use an assertion function to test say()
Assert::same('Hello John', $h->say('John'));

Thats' all!

Now we run tests from command-line using the tester command:

> tester
 _____ ___  ___ _____ ___  ___
|_   _/ __)( __/_   _/ __)| _ )
  |_| \___ /___) |_| \___ |_|_\  v2.5

PHP 8.2.0 | php -n | 8 threads
.
OK (1 tests, 0 skipped, 0.0 seconds)

Nette Tester prints dot for successful test, F for failed test and S when the test has been skipped.

Assertions

This table shows all assertions (class Assert means Tester\Assert):

  • Assert::same($expected, $actual) - Reports an error if $expected and $actual are not the same.
  • Assert::notSame($expected, $actual) - Reports an error if $expected and $actual are the same.
  • Assert::equal($expected, $actual) - Like same(), but identity of objects and the order of keys in the arrays are ignored.
  • Assert::notEqual($expected, $actual) - Like notSame(), but identity of objects and arrays order are ignored.
  • Assert::contains($needle, array $haystack) - Reports an error if $needle is not an element of $haystack.
  • Assert::contains($needle, string $haystack) - Reports an error if $needle is not a substring of $haystack.
  • Assert::notContains($needle, array $haystack) - Reports an error if $needle is an element of $haystack.
  • Assert::notContains($needle, string $haystack) - Reports an error if $needle is a substring of $haystack.
  • Assert::true($value) - Reports an error if $value is not true.
  • Assert::false($value) - Reports an error if $value is not false.
  • Assert::truthy($value) - Reports an error if $value is not truthy.
  • Assert::falsey($value) - Reports an error if $value is not falsey.
  • Assert::null($value) - Reports an error if $value is not null.
  • Assert::nan($value) - Reports an error if $value is not NAN.
  • Assert::type($type, $value) - Reports an error if the variable $value is not of PHP or class type $type.
  • Assert::exception($closure, $class, $message = null, $code = null) - Checks if the function throws exception.
  • Assert::error($closure, $level, $message = null) - Checks if the function $closure throws PHP warning/notice/error.
  • Assert::noError($closure) - Checks that the function $closure does not throw PHP warning/notice/error or exception.
  • Assert::match($pattern, $value) - Compares result using regular expression or mask.
  • Assert::matchFile($file, $value) - Compares result using regular expression or mask sorted in file.
  • Assert::count($count, $value) - Reports an error if number of items in $value is not $count.
  • Assert::with($objectOrClass, $closure) - Executes function that can access private and protected members of given object via $this.

Testing exceptions:

Assert::exception(function () {
	$h = new Greeting;
	$h->say(null);
}, InvalidArgumentException::class, 'Invalid name.');

Testing PHP errors, warnings or notices:

Assert::error(function () {
	$h = new Greeting;
	echo $h->abc;
}, E_NOTICE, 'Undefined property: Greeting::$abc');

Testing private access methods:

$h = new Greeting;
Assert::with($h, function () {
	// normalize() is internal private method.
	Assert::same('Hello David', $this->normalize('Hello david')); // $this is Greeting
});

Tips and features

Running unit tests manually is annoying, so let Nette Tester to watch your folder with code and automatically re-run tests whenever code is changed:

tester -w /my/source/codes

Running tests in parallel is very much faster and Nette Tester uses 8 threads as default. If you wish to run the tests in series use:

tester -j 1

How do you find code that is not yet tested? Use Code-Coverage Analysis. This feature requires you have installed Xdebug or PCOV extension, or you are using PHPDBG SAPI. This will generate nice HTML report in coverage.html.

tester . -c php.ini --coverage coverage.html --coverage-src /my/source/codes

We can load Nette Tester using Composer's autoloader. In this case it is important to setup Nette Tester environment:

require 'vendor/autoload.php';

Tester\Environment::setup();

We can also test HTML pages. Let the template engine generate HTML code or download existing page to $html variable. We will check whether the page contains form fields for username and password. The syntax is the same as the CSS selectors:

$dom = Tester\DomQuery::fromHtml($html);

Assert::true($dom->has('input[name="username"]'));
Assert::true($dom->has('input[name="password"]'));

For more inspiration see how Nette Tester tests itself.

Running tests

The command-line test runner can be invoked through the tester command (or php tester.php). Take a look at the command-line options:

> tester

Usage:
    tester [options] [<test file> | <directory>]...

Options:
    -p <path>                    Specify PHP interpreter to run (default: php).
    -c <path>                    Look for php.ini file (or look in directory) <path>.
    -C                           Use system-wide php.ini.
    -l | --log <path>            Write log to file <path>.
    -d <key=value>...            Define INI entry 'key' with value 'val'.
    -s                           Show information about skipped tests.
    --stop-on-fail               Stop execution upon the first failure.
    -j <num>                     Run <num> jobs in parallel (default: 8).
    -o <console|console-lines|tap|junit|none>
                                 Specify output format.
    -w | --watch <path>          Watch directory.
    -i | --info                  Show tests environment info and exit.
    --setup <path>               Script for runner setup.
    --temp <path>                Path to temporary directory. Default by sys_get_temp_dir().
    --colors [1|0]               Enable or disable colors.
    --coverage <path>            Generate code coverage report to file.
    --coverage-src <path>        Path to source code.
    -h | --help                  This help.

tester's People

Contributors

besanek avatar beutlin avatar dg avatar diegosardina avatar fprochazka avatar grogy avatar janbarasek avatar jantvrdik avatar jiripudil avatar juzna avatar kminekmatej avatar kravco avatar maddockman avatar majkl578 avatar martinmystikjonas avatar mesour avatar mikulas avatar milo avatar richard-ejem avatar skalda avatar smuuf avatar stekycz avatar tsusanka avatar ujovlado avatar vojtech-dobes avatar vrana avatar vrtak-cz avatar vysinsky avatar xmedeko avatar xpavp03 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tester's Issues

Add param to run a specific test method

I would like to run only specific method in my test case via cmd. I think, now this isn't possible because I can't get cli arguments in my bootstrap, where I have auto-run of my test cases.

Something like this:

> ./run-tests.sh -t testSomething Project/Test.phpt

... and in my bootstrap.php:

Tester\Environment::getArguments();

Of course, I can write own tester.php or specify name of method in my bootstrap.php (new TestCase()->runTest($name)) but this would by easier :)

CodeCoverage always appends data

CodeCoverage always appends data to coverage.dat file. But this file may exist from previous run and data may be outdated. In the result the coverage-reports generates nonsences.

Maybe related to #103.

Absolute paths in TAP output

Please, would it be possible to use absolute file paths in TAP output (diff line already uses them)? For NetBeans, it is not always possible to resolve them properly, like the first 2 files in the following output:

# in Tester/Framework/Assert.php(365)
# in Tester/Framework/Assert.php(57) Tester\Assert::fail()
# in nette-tester/tests/Greeting.test.phpt(16) Tester\Assert::same()

Thanks for considering it.

Wrong test counting with TestCase

Each test method should be count as test. Now Tester says you have run only 1 test when you run TestCase with multiple tests.

It is not good that after new test is written Tester output stays same as before.

NAN === NAN in PHP 5.3.1-4

In PHP 5.3.1 - 5.3.4 (maybe on Windows) is true, that

NAN == NAN
NAN === NAN

What to do:

  1. is should be fixed in Assert::same, notSame, equal and notEqual and perhaps elsewhere, but NAN is very marginal thing.

  2. in tests\Assert.equal.phpt should be removed test for NAN then PHP < 5.3.5

Assert::type - memory limit exhausted

I wrote a Nette presenter test where I tried to check if response was an instance of RedirectResponse. Unfortunately I have had a bug in my code and response type was TextResponse.

I would expect some friendly error message, but Nette Tester tries to dump asserted object into an error message. And when this object is complex structure like Nette Template is, memory limit can be exhausted.

Error message like "Expected type ...\RedirectResponse but ...\TextResponse given" would be better in this case.

P.S.: I think similar bugs may occur in other assertions too.

Runner: invoke output handlers after received SIGINT

8021168 doesn't work after @milo's changes, neither does commit 84247e2

vendor/nette/tester$ git co 8021168ed8c12d5b3f6547faf6ccb64e28ab0a16
Previous HEAD position was 84247e2... Runner: better SIGINT catching readability
HEAD is now at 8021168... Runner: invoke output handlers after received SIGINT (CTRL+C)
$ ./tests/run.sh -j 4 ./tests/
PHP 5.5.7 | 'php-cgi' -n -c 'tests/php.ini-unix' | 4 threads

FFF^C
$

Disable *.phpt scanning

Scanning subdirectories for tests is sometimes problem. Imagine the application structure:

application   (current working dir)
|--- tests
|--- vendor
|--- |--- nette
|--- |--- |--- nette
|--- |--- |--- tester

and when you run the Tester without args, vendor dir is scanned and Nette and Tester tests are found and executed. This can be supprise mainly for newbies. And it can leeds to problems with database tests.

In my opinion, the base problem is in git/composer that always clone whole repo, but it cannot be easily solved.

I would disable a Tester's automatic directory scan when not passed as arg. In that case, old behaviour can be enabled by dot:

Tester\tester .

No diff line in TAP output

Just found out that there is no diff line for simple assert:

Assert::same( 'Hello John', $h->say('Mary') );

(I am using the class from your readme.)

The TAP output is:

not ok nette-tester/tests/Greeting.test_1.phpt
# Failed: 'Hello Mary' should be 
#     ... 'Hello John'
# 
# in Tester/Framework/Assert.php(364) 
# in Tester/Framework/Assert.php(56) Tester\Assert::fail()
# in nette-tester/tests/Greeting.test_1.phpt(11) Tester\Assert::same()

Using latest Tester sources (cloned repo).

Thanks.

TAP format: more parser-friendly info about failure

Discussed in issue #39:

(2) could we provide more parser-friendly output for TAP for failures? Something like:

# expected: foo
# actual: bar

I get 2 lines for string comparison but only one line if I compare string and number:

# Failed: 'Hello JohnX' should be 
#     ... 'Hello John'
# Failed: 'Hello JohnX' should be 10

Thanks.

Simulate PHPUnit command line output? (enhancement)

Most of current IDE support PHPUnit as test tool with progress and success/errors list and other stuff.

My question is, how hard would be to implement CLI script simulating PHPUnit behavior?
Good enought would be just running tests and producing same output.
(eg.: like sendmail is template for most contemporary mailers)

Showing diff of long string

Please, use sth like

$prefix = strlen(Nette\Strings:findPrefix($expected, $actual));
$start = max($prefix - 5, 0);
echo substr($expected, $start), "does not match", substr($actual, $start);

btw, awesome coloring etc :-)

Show stderr in test output

Is there reason why tester shows only content of stdout of test execution after test failure?

I just spent about hour trying to figure out why my tests suddenly failing with no information at all, because causing exception was written to stderr.

It should be sufficient to append "2>&1" to Job command at Job.php:77

Or is there reason for not doing so?

Ensure that test has been really run

Sometimes can happend that none assertion is done in a test file, e.g. accidental die() before first assertion or just:

class MyTest extends TestCase {
    function testFail() {
        Assert::fail('FAIL');
    }
}

# and there is missing (new MyTest)->run();

... test seems to be passing.

First idea is to count Assert::method() calls, check it in the shutdown function and fail or skip the test. The plus can be nothing-saying statistics :)

This is not an issue :)

As an early adopter I am very curious in what phase is the development of this tool. Currently I'am using PHPUnit and Mockery for the unit testing. If I would be able to use the same tool as the framework I would be very happy.

Note: Currently the biggest pain for me are the final methods in the core framework classes. But that's another story.

When testing whole directory, only first file is tested

If I run Tester to test whole directory, only the first file in directory is tested.

I have folder named exampleTests containing two files ExampleTest.phpt and OtherTest.pthp. OtherTests.phpt is containing one failing test.
I run C:>php pathToTester\tester --tap -s -c php.ini pathToProject\exampleTests
Output is wrong, only ExampleTest.phpt is tested, all tests passed:
TAP version 13
ok tests\exampleTests\ExampleTest.phpt
1..1

I'm running on Win 7 64b, PHP 5.4.7, Nette Framework 2.0.12, Nette Tester v0.9.3

Thanks

Add "stable" and more detailed and continous reports

This issue is very similar to atoum/atoum#171.

We (NetBeans IDE) are currently thinking about nette/tester support [1] - but there are several things that would need to be done:

  • stable CLI output - please, consider adding CLI parameter porcelain (or similar) which would ensure that the CLI output will never change and can be easily parsed; something like e.g. Git has already
  • add more detailed CLI reports (maybe TAP format can be used). Perfect would be something like:
==> Suite MyApplicationSuite started
==> Test sayHello started
==> Test sayHello failure:
[information about failure - exception, error, failure]
==> Test sayHello finished: 5ms
==> Suite MyApplicationSuite finished: 25ms
==> Running duration: 0.15 second.

Thanks a lot.
[1] https://netbeans.org/bugzilla/show_bug.cgi?id=228679

Test which cannot be run in parallel

Is there any option how to tell that some tests should not run parallel?

I have some tests which on the setUp fill the database and on the tearDown restore original state of the DB. But when two tests run in parallel and they fill the same table they will fail because of unexpected data in the database.

It would be great if I can tell the runner, that the test should run only if there is no other test with the same flag (group or something else) active (currently running).

I know that there is option how to tell the runner that it should run only 1 thread, but I need only some tests to not run in parallel. Others can run normally, e.g. if they use different tables.

Feature: rename Assert to Check

as the PHP contains built-in function assert() the IDE's code completion is not so comfortable as it can be.

Assert

z3

Check

z4

Test previous exceptions

Sometime is useful to test previous exception too. Now it must be done by manual catching and walking throw previous. Would be nice to support:

Assert::throw(function() {
   throw new \Exception(..., ..., $previous);
}, '\Exception', 'message', '\Previous\Exception', 'Previous message');

or

Assert::throw(function() {
    throw new \Exception(..., ..., $previous);
  },
  array('\Exception', '\Previous\Exception'),
  array('message', 'Previous message')
);

or

Assert::throw(function() {
    throw new \Exception(..., ..., $previous);
  }, new \Exception('message', ..., new \PreviousException('Previous message'));

Cannot test Fatal errors

After commit Fatal errors cannot be hidden by @shutup (43a8b3d), the fatal errors cannot be tested by register_shutdown_function() because that one in Helpers::setup() exits and other is not executed.

It happens in Nette now with Debugger.shut-up.error.phpt (for example https://travis-ci.org/nette/nette/jobs/9130851).

Btw, during the investigation, I found, that PHP exit code is always 255 in case of Fatal error (maybe others too). ```php register_shutdown_function(function() { $error = error_get_last(); print_r($error); die(10); });

@missing();


php test.php; echo $?

</del>

test results of two methods in one test case may get mixed up

class SomeTest extends TestCase
{
    public function testAaa()
    {
        Assert::same(2, 2);
    }

    public function testBbb()
    {
        Assert::same(3, []);
    }
}

both fail with error array() should be 3 in testBbb()

only when runned with test runner, not directly

the reason is, that both test methods produce expected and actual files with the same name

Option fail-fast

Sometimes if I implement new features, I need to fix tests. When there is more failed tests, it becomes quite messy, so I would like to use some option like fail-fast, which will run tests and stops when some fails.

Fail if there is no assert in test

I wrote my first test with TestCase and I forgot to run the test case ( (new SomethingTestCase())->run(); ). This might be common mistake for former PHPUnit users.

Tester said that there was 1 passed test.

I thing that error or at least warning would be better if there is no assert in test.

Debugger does not identify the CLI environment

I am not sure if this belongs to the Debugger which causes the problem or to the Tester which runs the tests using the CGI version of PHP.

I enabled the Debugger in tests bootstrap to get a better error visualization and/or logging and I was surprised with the HTML output in the console:

<!DOCTYPE html>
   <meta charset="utf-8">
   <meta name=robots content=noindex>
   <meta name=generator content="Nette Framework">
   ...
   <title>Server Error</title>
   ...

With the DEVELOPMENT mode enabled there is of course the whole red-screen/debugger-bar...

I ve found the problem is in the isHtmlMode() method where the CLI mode is not successfully detected as the PHP_SAPI does not contain the string "cli" but "cgi-fcgi".

Am I missing sth or doing sth wrong?

If not it would be cool to add this value ("cgi-fcgi") to the condition too.

As I am running PHP in the FPM mode I was wondering if this value is not returned also in the server mode and it is not. There is the "fpm-cgi" string returned instead.

Error: File path 'php-cgi.exe' not found.

When running Nette Tester I got this error. The bug is somewhere in 0ab996a (previous commits works fine, this one and later do not).

Yes, php-cgi.exe is in PATH (otherwise the previous commits wouldn't work either).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.