JarCastingInstrumentor
Instrumentation toolkit built on top of Metrics
Status
Instrumenting Methods
This module's primary aim is to increase the ease with which you can monitor calls to the key methods in your java application. If you're not sure what methods to instrument or why you should be instrumenting anything in the first place, you should go check out Coda Hale's Metrics Introduction
In general, an instrumented method will report
- a count of calls in flight
- call rate
- timing
- error rate
Assuming we have the class
package com.mycompany;
class Example {
public void sayHello() {
System.out.println("Hello, World");
}
public static void main(String[] args) {
new Example().sayHello();
}
}
and we were to instrument the sayHello method, we'd get the following metrics registered and reported.
In Flight Count
We'll get a count of the current method calls in flight.
com.mycompany.Example.sayHello.inFlight.count-- a count of method calls currently being executed
Call Rate
We'll get several stats for how often the sayHello method is being called.
com.mycompany.Example.sayHello.count-- a lifetime count of calls to the methodcom.mycompany.Example.sayHello.mean_rate-- mean rate of callscom.mycompany.Example.sayHello.m1_rate-- rate of calls over the last 1 minutecom.mycompany.Example.sayHello.m5_rate-- rate of calls over the last 5 minutescom.mycompany.Example.sayHello.m15_rate-- rate of calls over the last 15 minutes
Timing Statistics
We'll get a Histogram of the timing of the method. That is, a statistical representation of the distribution of "How long did it take to call this method"
com.mycompany.Example.sayHello.max-- maximum time it took to call the methodcom.mycompany.Example.sayHello.mean-- mean time to call the methodcom.mycompany.Example.sayHello.min-- minimum time it took to call the methodcom.mycompany.Example.sayHello.p50-- time it took for 50% of calls to completecom.mycompany.Example.sayHello.p75-- time it took for 75% of calls to completecom.mycompany.Example.sayHello.p95-- time it took for 95% of calls to completecom.mycompany.Example.sayHello.p99-- time it took for 99% of calls to completecom.mycompany.Example.sayHello.p999-- time it took for 99.9% of calls to completecom.mycompany.Example.sayHello.stddev-- standard deviation of method call times
Error Rate
We'll get stats to track the rate at which the method throws exceptions
com.mycompany.Example.sayHello.errors.count-- a lifetime count of errorscom.mycompany.Example.sayHello.errors.mean_rate-- mean rate of errorscom.mycompany.Example.sayHello.errors.m1_rate-- rate of errors over the last 1 minutecom.mycompany.Example.sayHello.errors.m5_rate-- rate of errors over the last 5 minutescom.mycompany.Example.sayHello.errors.m15_rate-- rate of errors over the last 15 minutes
Percent Error Rate
We'll get stats to track the percentage of method calls that are throwing exceptions
com.mycompany.Example.sayHello.errors.total_pct-- a lifetime percent of errorscom.mycompany.Example.sayHello.errors.mean_pct-- mean percent of errorscom.mycompany.Example.sayHello.errors.m1_pct-- percent of errors over the last 1 minutecom.mycompany.Example.sayHello.errors.m5_pct-- percent of errors over the last 5 minutescom.mycompany.Example.sayHello.errors.m15_pct-- percent of errors over the last 15 minutes
HealthCheck
If you pass a threshold as the third argument, you'll get a HealthCheck that will monitor the percent of errors, and notify when the error rate exceeds the threshold.
Assuming the threshold given is 0.1 i.e. ten percent, and there were five errors in the last 100 calls, the healtcheck would report as "Healthy".
If there were, however, twenty errors in the last 100 calls, the healthcheck would report as "Unhealthy", with a message of "value=0.2&threshold=0.1"
HealthChecks can by logged by interrogating the underlying HealthCheckRegistry. The Metrics documentation has a good guide for getting value out of HealthChecks here.
The goal with healthchecks is to give us something that we can monitor to drive pagerduty/nagios/zabbix/whatever we're using for monitoring and alerting.
How to Instrument
Instrumenting with Instrumentor
The easiest way to instrument methods is using the Instrumentor class.
To get it, you'll need to include instrumentor-core:
<dependency>
<groupId>com.sproutsocial</groupId>
<artifactId>instrumentor-core</artifactId>
<version>0.7.5</version>
</dependency>
Continuing the example from earlier, lets instrument the sayHello method. To instrument methods that return void, you'll want to use Instrumentor#run.
To instrument the method, we'll need two things:
- an instance of
java.util.concurrent.Runnable - a name,
Stringwhich will be the prefix for the generated metrics.
Easier naming
In the example above, we used "com.mycompany.Example.sayHello" as the base name, but there's nothing special about the name, it can be any string.
If you're so inclined, you can use MetricRegistry.name(Class<?> klass, String... names) to name your method:
String baseName = MetricRegistry.name(Example.class, "sayHello");
// "com.mycompany.Example.sayHello"
package com.mycompany;
public class Example {
public void sayHello() {
System.out.println("Hello, World");
}
public static void main(String[] args) {
Example example = new Example();
String baseName = MetricRegistry.name(Example.class, "sayHello");
Instrumentor instrumentor = new Instrumentor();
Runnable runnable = new Runnable() {
public void run() {
example.sayHello();
}
};
// run the method
instrumentor.run(runnable, baseName);
}
}
With a lambda
With java 8 lambdas, the last few lines become a bit simpler.
Instead of
Runnable runnable = new Runnable() {
public void run() {
example.sayHello();
}
};
// run the method
instrumentor.run(runnable, baseName);
we can use
instrumentor.run(() -> example.sayHello(), baseName);
or even
instrumentor.run(example::sayHello, baseName);
Adding a healthcheck
To add a healthcheck, supply an error threshold (double) as the third argument.
instrumentor.run(example::sayHello, baseName, 0.1);
Instrumenting a callable
You can also instrument instances of java.until.concurrent.Callable:
package com.mycompany;
public class Example {
public String getGreeting() {
return "Hello, World";
}
public static void main(String[] args) {
Example example = new Example();
String baseName = MetricRegistry.name(Example.class, "sayHello");
Instrumentor instrumentor = new Instrumentor();
// run the method
String greeting = instrumentor.call(example::getGreeting, baseName); // "Hello, World"
}
}
Inspecting Results
The no-arg constructor for Instrumentor will create its own underlying MetricRegistry and HealthCheckRegistry. It exposes them through accessors so you can inspect the results.
package com.mycompany;
public class Example {
public void sayHello() {
System.out.println("Hello, World");
}
public static void main(String[] args) {
Example example = new Example();
String baseName = MetricRegistry.name(Example.class, "sayHello");
Instrumentor instrumentor = new Instrumentor();
// run the method
instrumentor.run(example::sayHello, baseName);
// get the MetricRegistry
MetricRegistry metricRegistry = instrumentor.getMetricRegistry();
Meter meter = metricRegistry.meter(baseName);
assert meter.getCount() > 0;
}
}
Supplying your own registries
If you want to supply your own registries, you can use Instrumentor.Builder
package com.mycompany;
public class Example {
public void sayHello() {
System.out.println("Hello, World");
}
public static void main(String[] args) {
Example example = new Example();
String baseName = MetricRegistry.name(Example.class, "sayHello");
// our own registries
MetricRegistry metricRegistry = new MetricRegistry();
HealthCheckRegistry healthCheckRegistry = new HealthCheckRegistry();
// pass into Instrumentor via Builder
Instrumentor instrumentor = Instrumentor.builder()
.metricRegistry(metricRegistry)
.healthCheckRegistry(healthCheckRegistry)
.build();
// run the method
instrumentor.run(example::sayHello, baseName);
Meter meter = metricRegistry.meter(baseName);
assert meter.getCount() > 0;
HealthCheck.Result result = healthCheckRegistry.runHealthCheck(baseName);
assert result.isHealthy();
}
}
Instrumenting with Guice AOP
There is also a module for Instrumenting using annotations and Guice AOP.
You'll need to include the instrumentor-aop module:
<dependency>
<groupId>com.sproutsocial</groupId>
<artifactId>instrumentor-aop</artifactId>
<version>0.7.5</version>
</dependency>
You can use the @Instrumented annotation with the InstrumentedAnnotations guice module to specify methods that should be instrumented. You can add the @Instrumented annotation to a class to instrument all of its public methods.
If you include an instance of the InstrumentedAnnotations module in your call to Guice.createInjector() (see below), then you can use the @Instrumented annotation to instrument methods. The module will also bind a MetricRegistry and a HealthCheckRegistry.
package com.mycompany;
class Example {
@Instrumented
public void sayHello() {
System.out.println("Hello, World");
}
public static void main(String[] args) {
Injector injector = Guice.createInjector(new InstrumentedAnnotations());
Example example = injector.getInstance(Example.class);
example.sayHello();
example.sayHello();
MetricRegistry metricRegistry = injector.getInstance(MetricRegistry.class);
Timer timer = metricRegistry.getTimers().get("com.mycompany.Example.sayHello");
assert timer.getOneMinuteRate() > 0;
Meter meter = metricRegistry.getMeters().get("com.mycompany.Example.sayHello.errors");
assert meter.getOneMinuteRate() == 0;
}
}
Method names
By default, the methods annotated by @Instrumented method will be named
com.somepackage.ClassName.methodName
But that can be overriden using the name attribute of Instrumented:
package com.mycompany;
class Example {
@Instrumented(name="mySweetName")
public void sayHello() {
System.out.println("Hello, World");
}
public static void main(String[] args) {
Injector injector = Guice.createInjector(new InstrumentedAnnotations());
Example example = injector.getInstance(Example.class);
example.sayHello();
example.sayHello();
MetricRegistry metricRegistry = injector.getInstance(MetricRegistry.class);
Timer timer = metricRegistry.getTimers().get("mySweetName");
assert timer.getOneMinuteRate() > 0;
Meter meter = metricRegistry.getMeters().get("mySweetName.errors");
assert meter.getOneMinuteRate() == 0;
}
}
Health Check
To add a healthcheck for the error percentage, just add an errorThreshold to the @Instrumented annotation.
package com.mycompany;
class Example {
@Instrumented(errorThreshold=0.1)
public void sayHello() {
System.out.println("Hello, World");
}
public static void main(String[] args) {
Injector injector = Guice.createInjector(new InstrumentedAnnotations());
Example example = injector.getInstance(Example.class);
example.sayHello();
example.sayHello();
HealthCheckRegistry healthCheckRegistry = injector.getInstance(HealthCheckRegistry.class);
HealthCheck.Result result = healthCheckRegistry.runHealthCheck(
"com.mycompany.Example.sayHello"
);
assert result.isHealthy();
}
}
Supplying your own registries
Like Instrumentor, InstrumentedAnnotations has a Builder that you can use to supply your own registries.
package com.mycompany;
class Example {
@Instrumented(errorThreshold=0.1)
public void sayHello() {
System.out.println("Hello, World");
}
public static void main(String[] args) {
// our own registries
MetricRegistry metricRegistry = new MetricRegistry();
HealthCheckRegistry healthCheckRegistry = new HealthCheckRegistry();
InstrumentedAnnotations annotationsModule = InstrumentedAnnotations.builder()
.metricRegistry(metricRegistry)
.healthCheckRegistry(healthCheckRegistry)
.build();
Injector injector = Guice.createInjector(annotationsModule);
Example example = injector.getInstance(Example.class);
example.sayHello();
example.sayHello();
Timer timer = metricRegistry.getTimers().get("mySweetName");
assert timer.getOneMinuteRate() > 0;
HealthCheck.Result result = healthCheckRegistry.runHealthCheck(
"com.mycompany.Example.sayHello"
);
assert result.isHealthy();
}
}
AOP Gotchas
Not seeing metrics that you think you should be? There are a couple of gotchas with guice AOP.
-
AOP only works for methods being called by other classes. This is because Guice AOP works by inserting a proxy in front of the instrumented class wherever it is injected. Calls of instrumented methods from within the same class will not go through this proxy.
-
AOP will only be applied to objects created by guice. If you just use
new Example(), the annotation will have no effect. -
You'll need to make sure that the registries used to build the
InstrumentedAnnotationsmodule are the same as the ones you're reading from.
Reporting Your Metrics
Now that you're tracking metrics, you'll want to report them.
If you're using Dropwizard, metrics reporting can be configured in your yaml config, and you can inspect metrics and healthchecks using the admin servlet.
To report to graphite/cloudwatch/etc see The Metrics Documentation.
You can also embed your own Admin Servlet.
