Month: July 2007

I just got back from OSCON, a conference on Open Source that O’Reilly hosts in Portland annually. The conference offered some interesting content and side-shows with some notable highlights (more on those in the next few days). Brendan and I gave a presentation on how a crew from Sun dropped in on Twitter to help them use DTrace to discover some nasty performance problems.

Here’s the presentation along with the D scripts and load generators we used for the talk.

Robert Scoble was kind enough to interview us last week for the ScobleShow. Robert pretty much let us riff continuously for half an hour — we clearly haven’t been getting to talk about DTrace enough lately. I thought he would trim it down a bit, but like a scene from Hard Boiled, it’s all there.

This picture captures my favorite moment (around 16:23) during the interview as the three of us try to formulate a connection between DTrace and green computing…

… and this is as good a time as any to plug the talk Brendan and I will be giving at the end of the month at OSCON. We’ll be talk about how team DTrace was able to solve some nasty Ruby scalability problems at Twitter; it’s in the Ruby track, but the principles apply for analysis of performance problems in all languages.

People often ask about the future direction of DTrace, and while we have some stuff planned for the core infrastructure, the future is really about extending DTrace’s scope into every language, protocol, and application with new providers — and this development is being done by many different members of the DTrace community. An important goal of this new work is to have consistent providers that work predictably. To that end, Brendan and I have started to sketch out an array of providers so that we can build a consistent model.

In that vein, I recently integrated a provider for our iSCSI target into Solaris Nevada (build 69, and it should be in a Solaris 10 update, but don’t ask me which one). It’s an USDT provider so the process ID is appended to the name; you can use * to avoid typing the PID of the iSCSI target daemon. Here are the probes with their arguments (some of the names are obvious; for others you might need to refer to the iSCSI spec):

The argument structures are defined as follows:

typedef struct conninfo {
string ci_local;        /* local host address */
string ci_remote;       /* remote host address */
string ci_protocol;     /* protocol (ipv4, ipv6, etc) */
} conninfo_t;
typedef struct iscsiinfo {
string ii_target;               /* target iqn */
string ii_initiator;            /* initiator iqn */
uint64_t ii_lun;                /* target logical unit number */
uint32_t ii_itt;                /* initiator task tag */
uint32_t ii_ttt;                /* target transfer tag */
uint32_t ii_cmdsn;              /* command sequence number */
uint32_t ii_statsn;             /* status sequence number */
uint32_t ii_datasn;             /* data sequence number */
uint32_t ii_datalen;            /* length of data payload */
uint32_t ii_flags;              /* probe-specific flags */
} iscsiinfo_t;
typedef struct iscsicmd {
uint64_t ic_len;        /* CDB length */
uint8_t *ic_cdb;        /* CDB data */
} iscsicmd_t;

Note that the arguments go from most generic (the connection for the application protocol) to most specific. As an aside, we’d like future protocol providers to make use of the conninfo_t so that one could write a simple script to see a table of frequent consumers for all protocols:

iscsi*:::,
http*:::,
cifs:::
{
@[args[0]->ci_remote] = count();
}

With the iSCSI provider you can quickly see which LUNs are most active:

iscsi*:::scsi-command
{
@[args[1]->ii_target] = count();
}

or the volume of data transmitted:

iscsi*:::data-send
{
@ = sum(args[1]->ii_datalen);
}

Brendan has been working on a bunch of iSCSI scripts — those are great for getting started examining iSCSI