bechte / junit-hierarchicalcontextrunner

This is a runner implementation that supports context hierarchies in JUnit. For more details please visit: https://github.com/bechte/junit-hierarchicalcontextrunner/wiki

junit-hierarchicalcontextrunner

This is a runner implementation that supports context hierarchies in JUnit. For more details please visit: https://github.com/bechte/junit-hierarchicalcontextrunner/wiki

Master Branch (Stable Releases):

Development Branch (Snapshot Releases):

junit-hierarchicalcontextrunner's Issues

Support for static inner classes

People have asked me, why the HierarchicalContextRunner does not support tests within static inner classes. This issue was created to start a discussion on this topic.

I was thinking about running tests for inner static classes as well, but I didn't see the point for doing so. They somehow break the hierarchical behavior, as they haven't direct access to the member variables (and even methods like the @Before/@after) of the enclosing class(es). Therefore, they are IMHO not much useful within the context hierarchy. I would rather have a new test class instead of an inner static class.

A typical use-cases I see for static inner class are stubs providing test-data for the tests or helpers. But I wouldn't like to have test in theses classes, would I?

Using the HierarchicalContextRunner it is unnecessary (and not wanted) to extend classes used to create a context, i.e. the inner member classes. Use inner member class definitions (public class X) as if they where context declarations (context X). Do not extend them.

I am looking forward to discussing with you. :)

Eclipse run single test

In Eclipse it is possible to select a single test method and execute only that one. But this functionality is not working with HierarchicalContextRunner. Always have initialzationError:

java.lang.Exception: The inner class ep.pericles.mrs.meeting.SaveCommentTest$Inserting$Update is not static.
at org.junit.runners.BlockJUnit4ClassRunner.validateNoNonStaticInnerClass(BlockJUnit4ClassRunner.java:113)
at org.junit.runners.BlockJUnit4ClassRunner.collectInitializationErrors(BlockJUnit4ClassRunner.java:102)
at org.junit.runners.ParentRunner.validate(ParentRunner.java:355)
at org.junit.runners.ParentRunner.<init>(ParentRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.<init>(BlockJUnit4ClassRunner.java:57)
at org.junit.internal.builders.JUnit4Builder.runnerForClass(JUnit4Builder.java:10)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
at org.junit.internal.requests.FilterRequest.getRunner(FilterRequest.java:31)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.<init>(JUnit4TestReference.java:33)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestMethodReference.<init>(JUnit4TestMethodReference.java:25)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader.createTest(JUnit4TestLoader.java:54)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader.loadTests(JUnit4TestLoader.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:452)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

It is clear that in Eclipse not the runner is used for execution test.But Is there any trick to resolve it?

Running a single test is useful when debugging code. Most of the time one test class has many test method which is working on the same production method. But when I am interested in a single scenario the noise of other executions (e.g.they are hitting the same breakpoint.....) are frustrating.

Release targeting final JUnit 4.12 release

When will a final version for final JUnit 4.12 release be released?

Background:
Today we updated JUnit in our project to the final 4.12 version. We also updated junit-hierarchicalcontextrunner to version 4.12.0-beta-1. Everythings seems to work fine :-).

But as the name suggests this release is targeted at the first beta of JUnit 4.12 and also declares JUnit 4.12-beta-1 as a dependency. I guess there wouldn't be any code changes for this release but still it would give us a better feeling to use a final release of junit-hierarchicalcontextrunner that officially targets the final release of JUnit 4.12 :-).

Inherited Inner test classes causes fails when concrete class has no tests

I wanted to inherit test and give each concrete class its own setup. This resulted in a exception that I included below. I've also included a code case. I am using Java 1.8. It seems HierarchicalContextRunner thinks that there is one more class than the really is.

@RunWith(HierarchicalContextRunner.class)
public class MyTest {
    public abstract class A {
        @Before
        public void setup() {}

        @Test
        public void runTestTwice() {}
    }

    public class B extends A {
        @Test
        public void runTestOnce() {}
    }

    public class C extends A {
    }
}

runTestTwice(org.mazarineblue.keyworddriven.MyTest$A) Time elapsed: 0.003 sec <<< ERROR!
java.lang.InstantiationException
at sun.reflect.InstantiationExceptionConstructorAccessorImpl.newInstance(InstantiationExceptionConstructorAccessorImpl.java:48)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at de.bechte.junit.runners.util.ReflectionUtil.createDeepInstance(ReflectionUtil.java:85)
at de.bechte.junit.runners.context.processing.MethodExecutor.run(MethodExecutor.java:76)
at de.bechte.junit.runners.context.processing.MethodExecutor.run(MethodExecutor.java:57)
at de.bechte.junit.runners.context.statements.RunChildren.evaluate(RunChildren.java:38)
at de.bechte.junit.runners.context.statements.RunAll.evaluate(RunAll.java:27)
....

Nesting Contexts multiple levels deep

In the spirit of RSpec's context (or Jasmine's describe()) Does this support multiple levels of nesting contexts? i.e.

-> Class (test fixture that represents the class/object level)
-> -> Method 1 (context)
-> -> -> Context 1 (context)
-> -> -> Context 2 (context)
-> -> Method 2 (context)

Android support

Hi,

It would be great if I would apply this structure to the Android instrumentation tests.

Thanks,

How do you run the tests?

Normally, I use the JUnit view in Eclipse. It shows the classes or @test methods, whcih can be Run-ed or Debug-ed. I can also unfold the nested class structure to reach the @test methods within a @RunWith(HierarchicalContextRunner) class. However, the Run/Debug menu is available for them but it fails with java.Exception: ParentClass$NestedClass is not static. The same error appears if I run the test method from the JavaEditor. How do you run your tests?

HierarchicalContextRunner for NUnit?

Is there an NUnit port of this runner? Or do you know of any similar projects for the NUnit framework?

Please provide support for JUnitParams

Hello,

I really appreciate your support for context base testing!
However in my tests I often use JUnitParamsRunner to parameterize test methods.
Please provide support for this runner so that we can use both facilities in parallel.
See also this question: http://stackoverflow.com/questions/21728702/using-junit-4-with-hierarchicalcontextrunner-and-junitparamsrunner

Automate Releases

A fix for #28 was merged into master 2 years ago, but it has not been incorporated into an official release. As a result, a fix for an issue in another project mockito/mockito#1578 is dependent on a dev snapshot of this library (I'm certain they will reject my fix for this very reason). As a developer that needs this fix, I can work around this by building master myself and releasing to my own Maven repository. It would be preferable however if releases were made automatically through some tool like Travis CI, especially since busy lives get in the way of doing this manually.

Compatibility with Infinitest plugin

Hi,

Have anyone tried to use this runner with the Infinitest plugin for Eclipse?

It actually runs the tests but for failing asserts it can't get the line code number (it always shows line 0 in the markers panel).

[Enhancement] [Hack] Running nested test

I was wondering if there was a possible way to run a single test inside a hierarchical context. This is probably a very hacky way of doing it and probably does not work for more than 1 nested class.

public class ExtendedHierarchicalContextRunner extends HierarchicalContextRunner {

    public ExtendedHierarchicalContextRunner(final Class<?> testClass) throws InitializationError {
        super(testClass);
    }

    /**
     * override statementBuilders to initialize enclosing Class' methods/rules 
     */
    protected void initialize() {
        super.initialize();
        statementBuilders = new ArrayList<>(statementBuilders);
        statementBuilders.add(0, (testClass, next, description, notifier) -> {
            if (testClass.isANonStaticInnerClass()) {
                Class<?> enclosingClass = testClass.getJavaClass().getEnclosingClass();
                if (enclosingClass == null)
                    return next;
                final List<FrameworkMethod> befores = new TestClass(enclosingClass).getAnnotatedMethods(BeforeClass.class);
                return befores.isEmpty() ? next : new RunBefores(next, befores, null);
            }
            return next;
        });
        statementBuilders.add(1, (testClass, next, description, notifier) -> {
            if (testClass.isANonStaticInnerClass()) {
                Class<?> enclosingClass = testClass.getJavaClass().getEnclosingClass();
                if (enclosingClass == null)
                    return next;
                final List<FrameworkMethod> afters = new TestClass(enclosingClass).getAnnotatedMethods(AfterClass.class);
                return afters.isEmpty() ? next : new RunBefores(next, afters, null);
            }
            return next;
        });
        statementBuilders.add(2, (testClass, next, description, notifier) -> {
            if (testClass.isANonStaticInnerClass()) {
                Class<?> enclosingClass = testClass.getJavaClass().getEnclosingClass();
                if (enclosingClass == null)
                    return next;
                final List<TestRule> classRules = new ArrayList<TestRule>();
                classRules.addAll(new TestClass(enclosingClass).getAnnotatedMethodValues(null, ClassRule.class, TestRule.class));
                classRules.addAll(new TestClass(enclosingClass).getAnnotatedFieldValues(null, ClassRule.class, TestRule.class));
                return classRules.isEmpty() ? next : new RunRules(next, classRules, description);
            }
            return next;
        });
        statementBuilders = Collections.unmodifiableList(statementBuilders);
    }
}

I am wondering if this is too hacky to be merged in or maybe can be refactored to work nicely in all situations.

Integrate 'HierachicalContextRunner' via a PR in the official JUnit repository

I think this will boost up the popularity of this awesome and very useful (custom) JUnit runner.

Usage of the HierarchicalContextRunner together with an ExpectedException rule makes all tests fail.

I tried to annotate the following class of the ilujo project's branch https://github.com/paschelino/ilujo/tree/path-simple-modifications with the HierarchicalContextRunner. Without the runner, all tests pass. With the runner, all tests fail:

https://github.com/paschelino/ilujo/blob/path-simple-modifications/src/test/java/de/cosmicsand/webtools/path/PathOperationsTest.java

Inherited Inner test classes do not run

I want to run the same hierarchical test set on two (independent) implementations of the same interface.

For this purpose I created an (abstract) unit test class with inner test classes that is inherited by two derived unit test classes.But when I execute the derived test classes, only the "top level" tests from the base class are executed.

Is this a bug, a limitation, or am I doing something wrong?

@Rules of outer class are not applied to the instance of the outer class

When using junit-hierarchicalcontextrunner together with Mockito's JUnit rule (e.g. MockitoJUnitRule or MockitoJUnit.rule() in newer versions) depending on where the mocks are defined (in the outer class or the inner class) some mocks aren't instantiated (cf. also mockito/mockito#353). I think this also applies to other JUnit rules that manipulate the test instance.

In the following example the @Mock annotated instance variable of type Runnerable is not instantiated (and the test fails with a NullpointerException):

@RunWith(HierarchicalContextRunner.class)
public class HierarchicalMockitoTest {
    @Rule
    public MockitoRule mockitoRule = MockitoJUnit.rule();

    @Mock
    private Runnable runnable;

    public class Context {
        @Test
        public void test() throws Exception {
            runnable.run();
        }
    }
}

When the mock is made an instance variable of HierarchicalMockitoTest$Context everything works as expected.

When debugging one sees that junit-hierarchicalcontextrunner correctly applies the rule onto the instance of HierarchicalMockitoTest$Context but not onto the instance of the outer class. I think the way junit-hierarchicalcontextrunner evalutes rules should be changed to also apply them to outer classes. But I wanted to start a discussion about it to see whether there are reasons against this change in behaviour. I could take a stab in case the change in behaviour is wanted.

Update to new library versions

Update from:

• org.mockito:mockito-all, 1.9.5 to 1.10.19
• junit:junit, 4.12-beta-1 to 4.12

Update the runner to implement Filterable

Our custom build environment allows us to select which tests to run by passing a regular expression on the command line. This is implemented via a JUnit Filter. It would be a huge help if your runner implemented Filterable.

Theories with HierarchicalContextRunner

Is it possible to use theories with the HierarchicalContextRunner?

In one of my junit tests classes all of my test cases are theories based using the Suite runner to mimic a "Context". Though I would like to use the HierarchicalContextRunner as it is exactly what I need. When I try to convert over I end up getting "No Context found", as it doesn't seem to pick theories.

Integration with Powermock Runner Delegator

Hi all,

due to a legacy untestable code, I gotta use a Powermock. I don't want to let your Runner apart, so I found out that PowerMock could work with a PowerMockRunnerDelegate in order to combine both runners.
The surprise was that doesn't work. I'll paste you below part of the exception:

org/junit/internal/runners/rules/RuleFieldValidator
java.lang.NoClassDefFoundError: org/junit/internal/runners/rules/RuleFieldValidator
    at de.bechte.junit.runners.validation.RuleValidator.<clinit>(RuleValidator.java:14)
    at de.bechte.junit.runners.context.HierarchicalContextRunner.getValidator(HierarchicalContextRunner.java:79)
    at de.bechte.junit.runners.context.HierarchicalContextRunner.validate(HierarchicalContextRunner.java:64)
    at de.bechte.junit.runners.context.HierarchicalContextRunner.<init>(HierarchicalContextRunner.java:58)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.powermock.reflect.internal.WhiteboxImpl.createInstance(WhiteboxImpl.java:1428)
    at org.powermock.reflect.internal.WhiteboxImpl.invokeConstructor(WhiteboxImpl.java:1301)
    at org.powermock.reflect.Whitebox.invokeConstructor(Whitebox.java:497)
    at org.powermock.modules.junit4.internal.impl.DelegatingPowerMockRunner$1.call(DelegatingPowerMockRunner.java:96)

From the current test implementation:

@RunWith(PowerMockRunner.class)
@PowerMockRunnerDelegate(HierarchicalContextRunner.class)
@PrepareForTest({Login.class})
public class LoginViaPasswordSpec extends SpecHelper {
...

}

Any hint/idea/approach to face that? I don't want to remove this runner... rspec is too deep in my mind.

Thanks in advanced,

Maven: declare testing dependencies with <scope>test</scope>

I just noticed that I have a dependency conflict between Mockito-1.9.5 and Mockito-1.9.5-RC1, which is caused by version 4.11.2 of the contextual runner:

mvn dependency:tree
...
[INFO] \- de.bechte.junit:junit-hierarchicalcontextrunner:jar:4.11.2:test
[INFO]    +- org.mockito:mockito-all:jar:1.9.5-rc1:test
[INFO]    \- org.hamcrest:hamcrest-all:jar:1.3:test

The version of Mockito in the master branch is now the latest 1.9.5, but the three testing dependencies are still in the default compile scope. Please set the dependencies to the test scope which will follow the best practice and avoid unwanted transitive inclusion.

Note: currently users of 4.11.2 whould be affected by the conflict with Mockito, so a release would be good?

Create new release from master

I use this library and need the fix for issue #28. It would be nice to have this fix in an official release of the master branch available in Maven Central. It would be great if you have the availability to do that. I am aware of the Release-HowTo wiki and I would do it myself but I'm afraid of making a mistake and damaging working released artifacts (it sounds like version 4.12.2, tag r4.12.2, new deployment version 4.12.3-SNAPSHOT is sufficient). If you can assure me the steps in the wiki are foolproof then I can create a new release for you. This is by no means an urgent request, I am able to continue on with what I need to do by building the project locally.

JDK 1.5 support

Is it possible to support the same JDK version as Junit itself?

I get an java.lang.NoClassDefFoundError: java/lang/ReflectiveOperationException using JDK 1.6.

As I can see ( https://github.com/junit-team/junit/blob/master/pom.xml ) the target JVM is 1.5 in JUnit. The class de.bechte.junit.runners.context.processing.MethodExecutor is using java.lang.ReflectiveOperationException which is part of 1.7.

Thx! Oliver

Inner classes marked @Ignore are not skipped

The @Ignore annotation works on the outer class (skipping all tests cases) and on methods (skipping individual test cases), but if an inner class is marked @Annotated, its test cases are not skipped.

Given:

@RunWith(HierarchicalContextRunner.class)
public class AppTest {
  public class RunContext {
    @Test
    public void shouldRun() throws Exception {
      assertTrue(true);
    }
  }
  @Ignore
  public class IgnoreClassContext{
    @Test
    public void shouldIgnore() throws Exception {
      assertTrue(false);
    }
  }
  public class IgnoreMethodContext{
    @Ignore
    @Test
    public void willIgnore() throws Exception {
      assertTrue(false);
    }
  }
}

I expect the result:

Tests run: 3, Failures: 0, Errors: 0, Skipped: 2

But instead, shouldIgnore() test fails, with the result:

Tests run: 3, Failures: 1, Errors: 0, Skipped: 1

Optimize Descriptions

The descriptions of the contexts and tests are somehow ugly. They contain all enclosing classes like EnclosingClass$SomeInnerClass$AnotherInnerClass. This doesn't read very well in the test result pane of the IDE. As the contexts are added as children to the EnclosingClass, their name should be only the simple name, e.g. SomeInnerClass and AnotherInnerClass in the given example. The test result should look like this:

• EnclosingClass
  • SomeInnerClass
    • AnotherInnerClass

Ordering

First of all, great work.

One minor issue: the ordering of Class.getDeclaredClasses() is not guaranteed and may change with different runtimes, so in your example, OldAccountContext could run before NewAccountContext, which would cause BankTest.BankContext .NewAccountContext.interestRateIsSet to fail.

Advice for implementing @IgnoreRest

I have a small JUnit extension called @IgnoreRest. It is like @Ignore but once a method is marked it is running only marked test and all the other is ignored.

More info:

• http://www.qualityontime.eu/articles/technology-stack/toolbox/junit-ignorerest/
• http://www.qualityontime.eu/articles/technology-stack/toolbox/junit-ignorerest2/

I have runner for JUnit and Spring (both was extremely easy to implement - second link) but when I had a look at hierarchicalcontextrunner I got a little bit confused.

Can you give me advice how to extend runner to support @IgnoreRest?

Thank you in advance.

Intellij running an individual test

Running Java 8, Intellij 14, junit 4.12. There appears to be a disconnect between intellij and junit for running a single test in a sub class. When I right click inside a method that's in a sub class I only see the option to run the whole outer test class. Its the same when using the Structure view.

Curious if there's a way to get intellij to recognize that the sub classes have runnable tests and to just run/debug that single sub test?

Baring that one case, everything else works fine.

Example:

@RunWith(HierarchicalContextRunner.class)
public class OuterClassTest {
    @Test
    public void testMe() {
        assertTrue(true); // right clicking here shows "Run testMe()"
    }

    public class SubTests {
        @Test
        public void testSub() {
            assertTrue(true); // right clicking here shows "Run OuterClassTest"
        }
    }
}