Adam Leventhal's blog

Search
Close this search box.

Month: May 2006

Jarod Jenson and I presented at JavaOne last week. If you’re interested, here are the slides we used:

Admittedly, a pdf can’t capture the scene of Jarod typing in DTrace commands while I futilely try to narrate, but you can run your own demos by checking out this, this, and this.


Technorati Tags:

Two of the DTrace creators (that’s me on the right, and Bryan on the left) apparently square off at the recent OpenSolaris user group meeting at JavaOne. You fill in the bubbles; the funniest response wins nothing but my heartfelt amusement.

At last year’s JavaOne, DTrace enjoyed some modicum of celebrity, being featured in a keynote, a session, and the DTrace challenge. The session was attended by about 1,000 people (out of about 10,000 at the conference), and the DTrace Challenge — which promised to find a performance win in Java applications brought to us or fork over an iPod — went an impressive 15 for 16 (and that one outlier was a trivial 200 line program — we’ve added a 2,000 line minimum this year).

Building on that success, Jarod Jenson and I are giving an updated session on DTrace for Java Thursday 5/18 2:45-3:45 in the Moscone Center Gateway 102/103. We’ll be giving a similar introduction to DTrace and to using DTrace on Java as at last year’s talk, and showing off some of the improvements we’ve made in the past year. Jarod’s also manning the DTrace Challenge again this year so get your Java application onto a USB key, CD or whatever, and bring it by booth 739; if he can’t make it go faster you’ll get an iPod, but if you were hoping for an iPod rather than a performance win prepare to be disappointed. Angelo Rajadurai and Peter Karlsson are also giving a hands on lab on using DTrace to examine Java applications (Friday 5/19 1:15-2:15 Moscone Center Hall E 130/131) so be sure to sign up for that if you want to get your hands dirty with the stuff we talk about at our session.


Technorati Tags:

As I’ve mentioned in the past, developers can add their own DTrace probes using the user-land statically defined tracing (USDT) mechanism. It’s been used to instrument Postgres and Apache, and to add observability into dynamic languages such as Java, Ruby, and php. I recently made a couple of improvements to USDT that I mentioned here and here, but I think deserve a little more discussion.

Adding USDT probes (as described in the DTrace manual) requires creating a file defining the probes, modifying the source code to identify those probe sites, and modifying the build process to invoke dtrace(1M) with the -G option which causes it to emit an object file which is then linked into the final binary. Bart wrote up a nice example of how to do this. The mechanisms are mostly the same, but have been tweaked a bit.

USDT in C++

One of the biggest impediments to using USDT was its (entirely understandable) enmity toward C++. Briefly, the problem was that the modifications to the source code used a structure that was incompatible with C++ (it turns out you can only extern "C" symbols at the file scope — go figure). To address this, I added a new -h option that creates a header file based on the probe definitions. Here’s what the new way looks like:

provider.d

provider database {
probe query__start(char *);
probe query__done(char *);
};

src.c or src.cxx

...
#include "provider.h"
...
static int
db_query(char *query, char *result, size_t size)
{
...
DATABASE_QUERY_START(query);
...
DATABASE_QUERY_DONE(result);
...
}

Here’s how you compile it:

$ dtrace -h -s provider.d
$ gcc -c src.c
$ dtrace -G -s provider.d src.o
$ gcc -o db provider.o src.o ...

If you’ve looked at the old USDT usage, the big differences are the creation and use of provider.h, and that we use the PROVIDER_PROBE() macro rather than the generic DTRACE_PROBE1() macro. In addition to working with C++, this has the added benefit that it engages the compiler’s type checking since the macros in the generated header file require the types specified in the provider definition.

Is-Enabled Probes

One of the tenets of DTrace is that the mere presence of probes can never slow down the system. We achieve this for USDT probes by only adding the overhead of a few no-op instructions. And while it’s mostly true that USDT probes induce no overhead, there are some cases where the overhead can actually be substantial. The actual probe site is as cheap as a no-op, but setting up the arguments to the probe can be expensive. This is especially true for dynamic languages where probe arguments such as the class or method name are often expensive to compute. As a result, some providers — the one for Ruby, for example — couldn’t be used in production due to the disabled probe effect.

To address this problem, Bryan and I came up with the idea of what — for lack of a better term — I call is-enabled probes. Every probe specified in the provider definition has an associated is-enabled macro (in addition to the actual probe macro). That macro is used to check if the DTrace probe is currently enabled so the program can then only do the work of computing the requisite arguments if they’re needed.

For comparison, Rich Lowe’s prototype Ruby provider basically looked like this:

rb_call(...
{
...
RUBY_ENTRY(rb_class2name(klass), rb_id2name(method));
...
RUBY_RETURN(rb_class2name(klass), rb_id2name(method));
...
}

Where rb_class2name() and rb_id2name perform quite expensive operations.

With is-enabled probes, Bryan was able to greatly reduce the overhead of the Ruby provider to essentially zero:

rb_call(...
{
...
if (RUBY_ENTRY_ENABLED())
RUBY_ENTRY(rb_class2name(klass), rb_id2name(method));
...
if (RUBY_RETURN_ENABLED())
RUBY_RETURN(rb_class2name(klass), rb_id2name(method));
...
}

When the source objects are post-processed by dtrace -G, each is-enabled site is turned into a simple move of 0 into the return value register (%eax, %rax, or %o0 depending on your ISA and bitness). When probes are disabled, we get to skip all the expensive argument setup; when a probe is enabled, the is-enabled site changes so that the return value register will have a 1. (It’s also worth noting that you can pull some compiler tricks to make sure that the program text for the uncommon case — probes enabled — is placed out of line.)

The obvious question is then “When should is-enabled probes be used?” As with so many performance questions the only answer is to measure both. If you can eke by without is-enabled probes, do that: is-enabled probes are incompatible with versions of Solaris earlier than Nevada build 38 and they incur a greater enabled probe effect. But if acceptable performance can only be attained by using is-enabled probes, that’s exactly where they were designed to be used.


Technorati Tags:

Recent Posts

April 17, 2024
January 13, 2024
December 29, 2023
February 12, 2017
December 18, 2016

Archives

Archives