JarCastingClaimant
| C L A I M |
| L E N S E |
| A N I L E |
| I S L E T |
| M E E T S |
Overview
By default, when a ScalaCheck property fails, you'll see its inputs (e.g. ARG_0: ...) but not the expression that actually failed. In many cases it is useful to be able to see the test or comparison that failed.
This library provides a Claim(...) macro which wraps any Boolean expression and converts it into a labeled Prop value. If the property fails, ScalaCheck will show you this label.
Quick Start
Claimant supports Scala 2.11, 2.12, and 2.13. It is available from Sonatype.
To include Claimant in your projects, you can use the following build.sbt snippet:
libraryDependencies += "org.typelevel" %% "claimant" % "0.1.0"
Claimant also supports Scala.js. To use Claimant in your Scala.js projects, include the following build.sbt snippet:
libraryDependencies += "org.typelevel" %%% "claimant" % "0.1.0"
Please note that Claimant is still a very young project. While we will try to keep basic source compatibility around the Claim(...) macro itself, it's very likely that Claim's library internals will change significantly between releases. No compatibility (binary or otherwise) is guaranteed at this point.
Examples
Here's an example of using Claim(...) to try to prove that Float is associative:
package mytest
import org.scalacheck.{Prop, Properties}
import org.typelevel.claimant.Claim
object MyTest extends Properties("MyTest") {
property("float is associative") =
Prop.forAll { (x: Float, y: Float, z: Float) =>
Claim((x + (y + z)) == ((x + y) + z))
}
}
Unfortunately for us, this isn't true and ScalaCheck will quickly find a counter-example:
[info] ! MyTest.float is associative: Falsified after 22 passed tests.
[info] > Labels of failing property:
[info] falsified: 0.2962196 == 0.29621956
[info] > ARG_0: 0.29622045
[info] > ARG_1: -8.811786E-7
[info] > ARG_2: 1.0369974E-8
The Claim(...) call inspects the expression and tries to determine what kind of operator is being used. Finding ==, it captures the left- and right-hand sides of that operator. Since the values are not equal, it labels the property with:
falsified: 0.2962196 == 0.29621956
This means that in addition to seeing which inputs cause a failure, we also see how much we failed by (around 4e-8 in this case).
Similarly, in some cases we want to be sure that at least one of several conditions is true. In this case, we want to be sure that either n is zero, or that n is not equal to -n.
package mytest
import org.scalacheck.{Prop, Properties}
import org.typelevel.claimant.Claim
object AnotherTest extends Properties("AnotherTest") {
property("ints have distinct inverses") = {
Prop.forAll { (n: Int) =>
Claim(n == 0 || n != -n)
}
}
}
Once again, we are out of luck! It turns out that Int.MinValue is its own negation (there is no positive value large enough to represent its actual negation). ScalaCheck helpfully shows us this:
[info] ! AnotherTest.ints have distinct inverses: Falsified after 0 passed tests.
[info] > Labels of failing property:
[info] falsified: (-2147483648 == 0 {false}) || (-2147483648 != -2147483648 {false})
[info] > ARG_0: -2147483648
In this case, Claim(_) helpfully shows us the how the different branches evaluate (summarizing each branch with {true} or {false}). Being able to see that the right branch ended up testing -2147483648 != -2147483648 cuts to the heart of the problem, and doesn't leave the caller guessing about how the conditions were evaluated.
Details
The Claim(_) macro recognizes many different kinds of Boolean expressions:
==and!=(universal equality)eqandne(referential equality)<,<=,>, and>=(comparisons)&&,&,||,|,^, and!(boolean operators)isEmptyandnonEmptystartsWithandendsWithcontains,containsSlice, andapplyisDefinedAt,sameElements, andsubsetOfexistsandforall(althoughFunction1values can't be displayed)Equiv#equiv
The Claim(_) macro also recognizes certain kinds of expressions which it will attempt to annotate, such as:
sizeandlengthcompare,compareTo, andlengthCompareminandmaxOrdering#compareandPartialOrdering#tryCompareOrdering.Implicits.infixOrderingOps
(For examples of the labels produced by these, see ClaimTest.)
It should be fairly straightforward to extend this to support other shapes, both for Boolean expressions and for general annotations.
Representation and formatting
Claimant uses its own Render[A] typeclass to produce human-readable representations of values. We get a number of benefits from this:
- We can provide a more useful representation of arrays than you would get with
.toString. - We can quote and escape
StringandCharvalues to make it easier to see their exact value. - We support user-provided display strategies for types they don't control.
- We support all of the above recursively, allowing collections, tuples, case classes, etc. to take advantage of all of these together.
For types that don't have their own Render instances, Claimant will use a low-priority implicit value to provide an implementation based on .toString. Some attempt has been made to provide implementations for most built-in Scala types, but PRs adding support for new instances (along with tests exercising them) will be gladly accepted.
In particular, Claimant makes it easy to define Render implementations for case classes. Simply do the following:
case class MyClass(...)
object MyClass {
implicit val renderForMyClass: Render[MyClass] =
Render.caseClass[MyClass]
}
The code above will define a render instance for MyClass which will render each of its fields with its corresponding instances. (In the future Claimant may use Shapeless to derive Render instances automatically.)
Purity
Currently Claim(...) will potentially evaluate its expression (or sub-expressions) multiple times, in any order. In the future we could be more uptight about preserving execution order and ensuring sub-expressions are run exactly as they would be, but so far this hasn't been a priority.
For example, assuming that the missilesFiredAt method is side-effecting, and returns the number of missiles that were just fired, consider the following code:
def missilesFiredAt(target: String): Int = {
val num = scala.util.Random.nextInt(3) + 3
println(s"firing $num missiles at $target")
num
}
property("notTooManyMissiles") =
Claim((missilesFiredAt("moon") max missilesFiredAt("mars")) < 4)
Setting aside the questionable wisdom of launching missiles during a test, here's an example of the output we might see:
firing 5 missiles at moon
firing 3 missiles at mars
firing 4 missiles at moon
firing 4 missiles at mars
firing 4 missiles at moon
firing 3 missiles at mars
[info] ! MissileTest.notTooManyMissiles: Falsified after 0 passed tests.
[info] > Labels of failing property:
[info] falsified: 4 max 4 {4} < 4
As we can see, Claimant is evaluating each expression multiple times. The values we see in the test label (4 for the Moon and 4 for Mars) aren't necessarily the same ones used to describe the test failing, we also see that at various points we launched 5 missiles at the Moon, and 3 missiles at Mars.
In cases where side-effects are unavoidable, consider evaluating them before calling Claim(...):
property("notTooManyMissiles") = {
val x = missilesFiredAt("moon")
val y = missilesFiredAt("mars")
Claim((x max y) < 4)
}
This will result in more consistent test output:
firing 5 missiles at moon
firing 5 missiles at mars
[info] ! MissileTest.notTooManyMissiles: Falsified after 0 passed tests.
[info] > Labels of failing property:
[info] falsified: 5 max 5 {5} < 4
Limitations
Currently Claim(...) only expands a set of known methods. This means that if you have methods which return Boolean and write something like Claim(Verifier.verify(dataSet)) your test failures will look something like this:
[info] ! FancyTest.verify data sets: Falsified after 4 passed tests.
[info] > Labels of failing property:
[info] falsified: false
[info] > ARG_0: DataSet(...)
The ways to fix this are:
- Have
verifyreturn a richer result. - Inline the
verifylogic in theClaim(...)call. - Extend Claimant to support
Verifier.verify.
Another problem is that Claim(...) inspects method calls based on their AST shape. This means that type application, implicit parameters, etc. need to be explicitly supported. This also means that implicit enrichment (or bedazzlement) can muddy the waters a bit and obscure the underlying values.
(For an example of how to deal with enrichment, see the support for Ordering.Implicits.infixOrderingOps.)
Development
To measure code coverage for 2.12, do the following:
$ sbt '++ 2.12.12' coreJVM/clean coverage coreJVM/test coverageReport
Assuming everything works, the result should end up someplace like:
.jvm/target/scala-2.12/scoverage-report/index.html
To measure coverage in 2.11 you'd instead do:
$ sbt '++ 2.11.12' coreJVM/clean coverage coreJVM/test coverageReport
And the result would end up someplace like:
.jvm/target/scala-2.11/scoverage-report/index.html
There's at least some code in Claimant that is specific to either 2.11 or 2.12, making it unlikely that we'll achieve 100% coverage under either version independently.
Future Work
There are a ton of possible improvements:
- Support more methods/shapes.
- Minimize recomputation in the macro.
- Consider using raw trees instead of quasi-quotes.
- Consider supporting fancy diagrams
- Consider supporting color output
- Consider an extensible/modular design
Community
People are expected to follow the Scala Code of Conduct when discussing Claimant on GitHub, the Gitter channel, or other venues.
Credits
This library was inspired by the assert(...) macro found in ScalaTest.
Copyright and License
All code is available to you under the Apache 2 license, available at https://opensource.org/licenses/Apache-2.0.
Copyright Erik Osheim, 2019.