Several months ago I put up a tutorial on using Jetty, Jersey, Jackson and Guice to make a small JSON web service stack. The code is more of a proof of concept than anything else, but we’re going to be giving it a few more real-world features today by adding the use of the Metrics library.
As a quick review, the project is a web service that allows accessing peanut butter & jelly sandwiches over HTTP. It has a primitive “stats” resource to allow you to see how many sandwiches have been made and how much peanut butter & jelly was used. This stats facility is more of demo of how to wire stuff together with Guice and how requests are routed to JAX-RS resources than a useful way to monitor usage, though.
We’re going to do two things to improve the situation. First, we’ll re-implement the number of sandwiches and amount of PB&J tracking using individual Counters and Gauges. Second, we’ll add code to automatically create various useful metrics for every resource method via Jersey ResourceFilter, ResourceFilterFactory, ContainerFilterResponseFilter, ResourceMethodDispatchAdapter, and ResourceMethodDispatchProvider implementations. Fun! :)
As before, I’ll outline the code changes in this article, but the complete code is in the repo for your browsing pleasure.
Reimplementing Sandwich stats
The SandwichStats object (a singleton) keeps track of the number of sandwiches and the amount of PB & J used to make them. This data is exposed through SandwichStatsResource, which response to GET of /sandwich/stats with data like {"sandwichesMade":2,"gramsOfJam":200,"gramsOfPeanutButter":400}. We want to keep track of those three numbers with metrics now as well. Metrics (the library) offers many different types of metrics (the concept). For these three numbers, there are two ways we could do it. We could use a Gauge to expose the numbers that SandwichStats is already tracking internally, or we could use a Counter that we increment whenever we increment the fields that SandwichStats already has. To demonstrate the use of Gauges and Counters, we’ll do some of both.
When creating a metric object, you can use a MetricsRegistry instance or the static method on the Metrics class. If you have multiple applications running inside the same JVM, or need that separation of metrics for other reasons, use MetricsRegistery. If you aren’t doing that, you can use the static methods on Metrics, but we’ll go ahead and use a MetricsRegistry because it’s just as easy as using the Metrics class when we have Guice to help us.
First, add a binding for MetricsRegistry in a Guice module:
bind(MetricsRegistry.class).toInstance(Metrics.defaultRegistry());
We’ll inject a MetricsRegistry into the SandwichStats ctor:
private final Counter jamCounter; private final Counter pbCounter; @Inject SandwichStats(MetricsRegistry metricsRegistry) { metricsRegistry.newGauge(SandwichStats.class, "sandwich-count", new Gauge<Integer>() { @Override public Integer value() { return sandwichesMade; } }); jamCounter = metricsRegistry.newCounter(SandwichStats.class, "grams-of-jam"); pbCounter = metricsRegistry.newCounter(SandwichStats.class, "grams-of-pb"); }
Then, we can update the existing recordSandwich method:
synchronized void recordSandwich(Sandwich sandwich) { sandwichesMade++; gramsOfJam += sandwich.getGramsOfJam(); gramsOfPeanutButter += sandwich.getGramsOfPeanutButter(); jamCounter.inc(sandwich.getGramsOfJam()); pbCounter.inc(sandwich.getGramsOfPeanutButter()); }
We don’t have to do anything for the “sandwich-count” metric because it’s a Gauge that references the existing field. The other two metrics are Counters, and therefore need to be updated separately.
Start up the service’s main method and open JConsole. When you connect to your process, open the MBeans tab and look in the com.teamlazerbeez.http.sandwich package. You should see some SandwichStats MBeans. Expand the grams-of-jam MBean, select the attribute and double click the numeric value to open a graph. Re-run the /sandwich/create request a few times and note that the graph will periodically update.
Automatic Jersey Request Metrics
Using and creating metrics by hand is already useful, but we can do better. We’re going to hook into the Jersey request dispatch mechanism to automatically create metrics for per-status code counts, etc. for every JAX-RS resource method.
In Jersey, resource methods are represented by subclasses of AbstractMethod. AbstractMethod is an abstract class representing a method that is invoked by Jersey. This could be a setter that’s invoked by Jersey’s IoC mechanism or a resource method or other types of methods. We only care about two subclasses: AbstractResourceMethod and AbstractSubResourceMethod (a subclass of AbstractResourceMethod). In JAX-RS, a resource method is one that uses the @Path annotation of its containing class, and a sub-resource method is one that has a @Path annotation of its own.
There are two mechanisms that we’re going to use in the Jersey request invocation pipeline to do this. We’ll do the the simpler one first.
Per-resource method counters with ResourceFilterFactory
Think of a Jersey ResourceFilter as filling a subset of the role of a javax.servlet.Filter. It can observe and modify the input and the output of a JAX-RS method invocation, but it does not wrap the actual invocation the way a servlet Filter does. This is still useful, though, since we can observe the HTTP status codes being output and create metrics to expose them.
A ResourceFilter itself doesn’t do much of anything except provide getters for ContainerRequestFilter and ContainerResponseFilter. Those two do the actual work. In our case (observing status codes) we only care about the latter. However, ResourceFilter objects are created by ResourceFilterFactory implementations, so we’ll start there. Here’s part of an implementation of a ResourceFilterFactory (check the source for the full details):
@Override public List<ResourceFilter> create(AbstractMethod am) { // documented to only be AbstractSubResourceLocator, AbstractResourceMethod, or AbstractSubResourceMethod if (am instanceof AbstractSubResourceLocator) { // not actually invoked per request, nothing to do logger.debug("Ignoring AbstractSubResourceLocator " + am); return null; } else if (am instanceof AbstractResourceMethod) { String metricBaseName = getMetricBaseName((AbstractResourceMethod) am); Class<?> resourceClass = am.getResource().getResourceClass(); return Lists.<ResourceFilter>newArrayList( new HttpStatusCodeMetricResourceFilter(metricBaseName, resourceClass)); } else { logger.warn("Got an unexpected instance of " + am.getClass().getName() + ": " + am); return null; } }
What we see here is that if the AbstractMethod is an AbstractResourceMethod, we generate a ‘metricBaseName’ (a prefix common to all metrics tied to that resource method) and return a list of ResourceFilter objects containing a HttpStatusCodeMetricResourceFilter, which is one of our classes. Here’s what that class does (again, check the source).
final class HttpStatusCodeMetricResourceFilter implements ResourceFilter, ContainerResponseFilter { .... @Override public ContainerResponse filter(ContainerRequest request, ContainerResponse response) { Integer status = response.getStatus(); Counter counter = counters.get(status); if (counter == null) { // despite the method name, this actually will return a previously created metric with the same name Counter newCounter = metricsRegistry.newCounter(resourceClass, metricBaseName + " " + status + " counter"); Counter otherCounter = counters.putIfAbsent(status, newCounter); if (otherCounter != null) { // we lost the race to set that counter, but shouldn't create a duplicate since Metrics.newCounter will do the right thing counter = otherCounter; } else { counter = newCounter; } } counter.inc(); return response; } }
This filter doesn’t modify the response; instead, it just gets the HTTP status, gets the appropriate Counter (creating it if need be), and increments it.
We have enough now to be able to hook our status-code-counting filter into Jersey. This is a little ungainly, but basically we’re informing GuiceContainer (the Jersey code that interacts with Guice and Guice Servlet) the ResourceFilterFactory to use. You can specify more than one (comma-separated strings, Class[], etc — check the docs on ResourceConfig.PROPERTY_RESOURCE_FILTER_FACTORIES) but we have just one here.
Before:
serve("/*").with(GuiceContainer.class);
After:
Map<String, String> guiceContainerConfig = new HashMap<String, String>(); guiceContainerConfig.put(ResourceConfig.PROPERTY_RESOURCE_FILTER_FACTORIES, HttpStatusCodeMetricResourceFilterFactory.class.getCanonicalName()); serve("/*").with(GuiceContainer.class, guiceContainerConfig);
Start up the service and check the MBeans. You’ll see SandwichStats, but nothing for the SandwichMakerResource until you use the “/sandwich/create” endpoint. Once you do that, you’ll see a metric named “/sandwich/create GET 200 counter” appear, and it will update appropriately as you make more requests.
Per-resource method timing with ResourceMethodDispatchAdapter and friends
Status code counters are great, but we really want to be able to generate timing info for request invocation, and to get timing information we’d have to use a ThreadLocal in a ContainerRequestFilter/ContainerResponseFilter pair, and that’s not how we roll.
The Metrics library has built-in support for timing method invocations if you annotate the methods you wish to be timed with @Timed. This uses Guice AOP under the hood, which has a few limitations (no non-final classes, must instantiate via Guice, etc.), so let’s see if we can do it a little less invasively.
We’ll hook into the Jersey method invocation mechanism via ResourceMethodDispatchAdapter and ResourceMethodDispatchProvider. A ResourceMethodDispatchProvider is responsible for creating RequestDispatcher objects for AbstractResourceMethods, and a ResourceMethodDispatchAdapter is a Jersey extension mechanism to allow us to control what ResourceMethodDispatchProvider gets used. Therefore, by registering our ResourceMethodDispatchAdapter, we can wrap the default ResourceMethodDispatchProvider with one that gathers timing info. The code you’ll see here does timing for all resource methods, but you could trivially make it conditional on an annotation on the resource method or class or anything you wish.
We’ll start with our custom RequestDispatcher implementation TimingRequestDispatcher. We want to use whatever dispatcher Jersey provides under the hood, so we wrap that dispatcher and just capture timing info.
@Override public void dispatch(Object resource, HttpContext context) { final TimerContext time = timer.time(); try { wrappedDispatcher.dispatch(resource, context); } finally { time.stop(); } }
We then can use this RequestDispatcher in TimingResourceMethodDispatchProvider. If you wanted to have conditional logic to sometimes not create timers, this would be a good place to do it.
@Override public RequestDispatcher create(AbstractResourceMethod abstractResourceMethod) { String metricBaseName = getMetricBaseName(abstractResourceMethod); Timer timer = metricsRegistry.newTimer(abstractResourceMethod.getResource().getResourceClass(), metricBaseName + " timer"); return new TimingRequestDispatcher(wrappedProvider.create(abstractResourceMethod), timer); }
We can use that ResourceMethodDispatchProvider in a TimingResourceMethodDispatchAdapter.
@Override public ResourceMethodDispatchProvider adapt(ResourceMethodDispatchProvider provider) { return new TimingResourceMethodDispatchProvider(metricsRegistry, provider); }
And then bind your Adapter class (which must also be annotated with com.google.inject.Singleton and javax.ws.rs.ext.Provider):
bind(TimingResourceMethodDispatchAdapter.class);
Start up the service and you should see Timer metrics being generated for all endpoints. It’s a little bit more convoluted to use the ResourceMethodDispatchAdapter approach, but it’s probably only really needed for timing; the ResourceFilterFactory approach should work for most everything else. Either way, now you know how to use the basics of the Metrics library, and how to execute custom logic before (ContainerRequestFilter), after (ContainerResponseFilter), and around (ResourceMethodDispatchAdapter) Jersey resource method invocation.