Adam Leventhal's Blog

DTrace User-Land

One of the primary motivations for DTrace was the absence of a framework that united observability into all aspects of the system. There were certainly tools for looking at the individual components (iostat(1) for I/O; mpstat(1) and prstat(1) for some basic system monitoring; truss(1), gdb(1), mdb(1) and dbx(1) for examining processes), but correlating the data from the disparate sources was difficult or impossible (tell that to a room of system administrators and they start nodding like a bunch of Barry Bonds bobble-heads). DTrace was designed to fill that hole -- to be a framework for systemic analysis and to provide a single interface for observing and combining data from any corner of the system.

I joined the DTrace team because I had some ideas for how to implement user-land tracing (which I've discussed ad nauseum here, here, here, and pretty much the rest of my blog). At that point, most of the work on DTrace had been around the framework and the kernel function boundary tracing provider (fbt). The kernel tracing providers were -- and continue to be -- immensely useful for us working on the Solaris kernel, but when I talk to customers and folks from Sun working to get performance wins on business applications, they're using the user-level tracing components for a large majority of the D-scripts they write. But that's not to say that the kernel tracing components aren't vital -- without facilities like the io provider or the sched provider many of those performance wins would be impossible to attain -- rather most of the problems manifest themselves in user-land and that's where most investigations begin.

When I (or others) start using DTrace on an application, the first D invocations we use are intended to gather information about the application -- where is it spending its time, what are the hot functions, etc. As the investigation narrows in scope further and further, it becomes increasingly important to understand the underlying activity in the kernel be it the many kernel function calls, or its interaction with the system's physical resources. The user-level and kernel providers are both required to form a complete whole. Without either half of the equation, you aren't left with a DTrace that's half as useful; you're left with what is effectively another specialized tool for a specific area of the system that doesn't actually address the problem of systemic comprehension.

We've continued working to improve DTrace since we first integrated it into Solaris 10 -- they're have been over 200 changes to add features, or address problems. It's worth noting that much of our recent work and the bulk of our future work involves improving support for user-land. DTrace tooks its first steps into the JVM just a few months ago, and there's a huge amount of work left to be done for truly elegant integration. User-level statically defined tracing (USDT) has proved to be a great tool, and we're working to make it easier to support, maintain and use.

There's still some work to be done in terms of coverage for kernel tracing -- a more general networking provider would be great -- but in many ways, tracing the kernel was the easy part. In kernel-mode we have nearly complete control of the hardware and the software; there's only one kernel to worry about and it can't go away (or if it does DTrace goes with it). In user-land processes can come and go as they please; they can be as ill-constructed, ill-compiled, or ill-behaved as they like; they can have complex interactions between each other; and depend on wildly unpredictable behaviors of the system. In some ways user-land is simpler in that you can't destroy the box, but as long as they obey a few rules, applications can be as crazy as they like. Taming user-land continues to be an ongoing challenge for DTrace.

What has made DTrace so successful is not its ability to observe any one aspect of the system (be it I/O, Java, or C applications), but its ability to observe the whole system. Since the framework lives in the kernel, the kernel was the first obvious source of data, but without user-level tracing DTrace would be an incomplete solution to understanding the system as a whole.

Technorati Tags: DTrace Solaris OpenSolaris