JarCastingmultitests
JUnit 4 rule and annotations to multiple test execution: repeat, retry and run in parallel: @Repeat, @Retry and @Parallel respectively.
Usage
Declaring dependency
Gradle
In dependencies section of build.gradle file add test dependency on junit4 and com.github.rrrekin:multitests-junit4:0.7.0:
dependencies {
compile ......
testCompile 'junit:junit:4.11'
testCompile 'com.github.rrrekin:multitests-junit4:0.7.0'
}
Maven
in the dependency section of pom.xml file add dependency on junit4 and multitests-junit4:
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.github.rrrekin</groupId>
<artifactId>multitests-junit4</artifactId>
<version>0.7.0</version>
<scope>test</scope>
</dependency>
</dependencies>
Using in test class
Sample usage is included in tests files RetryTest.java, RepeatTest.java and Parallel.java.
To enable annotations following rule has to be added to JUnit4 test class:
@Rule public MultiTestsRule multiTests = new MultiTestsRule();
@Retry annotation
This annotation can be used for tests that for some reason tend to fail sometimes for a reason outside our control - dependency on external system, network problems, etc.
By default method annotated with this annotation is attempted to execute maximum 3 times. This can be changed by the value provided as an annotation argument.
Examples of usage:
public class RetryTest {
private static int counter1 = 0;
private static int counter3 = 0;
@Rule public MultiTestsRule multiTests = new MultiTestsRule();
@Test
@Retry()
public void test1() throws Exception {
assertTrue(counter1++ % 3 == 2);
}
@Ignore("Will fail anyway")
@Test()
@Retry(7)
public void test2() throws Exception {
assertTrue(false);
}
@Test
@Retry(10)
public void test3() throws Exception {
assertTrue(counter3++ % 7 == 0);
}
}
test1will succeed after retriestest2cannot succeed, so it will fail anywaytest3will succeed, as the number of retries is increased to 10
@Repeat annotation
This annotation can be used for tests, that sometimes can succeed, and to increase chance of failure, the test should be repeated multiple times.
Examples of usage:
public class RepeatTest {
private static int counter1 = 0;
private static int counter2 = 0;
private static int counter3 = 0;
@Rule public MultiTestsRule multiTests = new MultiTestsRule();
@Test
@Repeat()
public void test1() throws Exception {
counter1++;
assertTrue("Fails for " + counter1, counter1 < 11);
}
@Test()
@Repeat(6) // Will fail for 7th time
public void test2() throws Exception {
counter2++;
assertTrue("Fails for " + counter2, counter2 < 7);
}
@Ignore("Will fail")
@Test
@Repeat(25)
public void test3() throws Exception {
assertFalse("Fails for " + counter3, counter3++ % 7 == 6);
}
}
Tests 1 & 2 above succeed, but will fail if the number of repetition will be increased. Test 3 will fail as the increased number of repetitions will lead to failure.
@Parallel annotation
This annotation can be helpful for testing thread-safety. It causes to start multiple threads (default 10), that are synchronized on cyclic barrier to synchronously start test method code execution in all threads. Test execution is stopped and test is failed when timeout is reached. The default timeout value is 10000 milliseconds.
Examples of usage:
public class ParallelTest {
private static final List<String> list1 = new ArrayList<String>();
private static final List<String> list2 = new CopyOnWriteArrayList<String>();
private static final List<String> list3 = new CopyOnWriteArrayList<String>();
@Rule public MultiTestsRule multiTests = new MultiTestsRule();
@Ignore("Fails with ConcurrentModificationException")
@Test
@Parallel(100)
public void test1() throws Exception {
final StringBuilder concatenation = new StringBuilder();
for(final String str: list1){
concatenation.append(str);
}
list1.add("abc");
}
@Test()
@Parallel(100)
public void test2() throws Exception {
final StringBuilder concatenation = new StringBuilder();
for(final String str: list2){
concatenation.append(str);
}
list2.add("abc");
}
@Ignore("Will fail due to timeout")
@Test
@Parallel(timeout = 14)
public void test3() throws Exception {
final StringBuilder concatenation = new StringBuilder();
for(final String str: list3){
Thread.sleep(15);
concatenation.append(str);
}
list3.add("abc");
}
}
test1fails becauseConcurrentModificationExceptionis thrown, as ArrayList is not thread-safetest2succeeds as it uses thread-safe CopyOnWriteArrayListtest3fails as the timeout is set to 14ms, but the test method contains sleep fot 15ms
Interactions with test runner
It is important to understand how multiple executions are performed.
Class setup and teardown (@BeforeClass & @AfterClass)
Executed as usual before and after all test methods.
Test method setup and teardown (@Before & @After)
Executed before and after any repeated method execution, so e.g. for @Repeat(42), both setup and teardown methods will be executed 42 times.
Static fields
Static fields are behaving as expected - they are initialized once at class startup, and keep changes from methods all the time.
Non-static fields
Non static fields are behaving a little bit strange when they are not initialized in the @Before methods, but in field declaration. They are initialized once per method, so when they are e.g. set to 0, and then incremented in method annotated with @Repeat, they are incremented to 10 (or other number when non-default number of repetitions is used). But when the field is reinitialized in the @Before method, it is always incremented once.
Compilation
Clone the source code repository:
git clone https://github.com/rrrekin/multitests.git
cd multitests
Execute gredle wrapper with build task:
./gradlew build
Complied jar file is located in build/libs/ directory.