Adam Leventhal's blog

Month: June 2008

Jonathan had a terrific post yesterday that does an excellent job of presenting Sun’s strategy for flash for the next few years. With my colleagues at Fishworks, an advanced product development team, I’ve spent more than a year working with flash and figuring out ways to integrate flash into ZFS, the storage hierarchy, and our future storage products — a fact to which John Fowler, EVP of storage, alluded recently. Flash opens surprising new vistas; it’s exciting to see Sun leading in this field, and it’s frankly exciting to be part of it.

Jonathan’s post sketches out some of the basic ideas on how we’re going to be integrating flash into ZFS to create what we call hybrid storage pools that combine flash with conventional (cheap) disks to create an aggregate that’s cost-effective, power-efficient, and high-performing by capitalizing on the strengths of the component technologies (not unlike a hybrid car). We presented some early results at IDF which has already been getting a bit of buzz. Next month I have an article in Communications of the ACM that provides many more details on what exactly a hybrid pool is and how exactly it works. I’ve pulled out some excerpts from that article and included them below as a teaser and will be sure to post an update when the full article is available in print and online.

While its prospects are tantalizing, the challenge is to find uses for flash that strike the right balance of cost and performance. Flash should be viewed not as a replacement for existing storage, but rather as a means to enhance it. Conventional storage systems mix dynamic memory (DRAM) and hard drives; flash is interesting because it falls in a sweet spot between those two components for both cost and performance in that flash is significantly cheaper and denser than DRAM and also significantly faster than disk. Flash accordingly can augment the system to form a new tier in the storage hierarchy – perhaps the most significant new tier since the introduction of the disk drive with RAMAC in 1956.

A brute force solution to improve latency is to simply spin the platters faster to reduce rotational latency, using 15k RPM drives rather than 10k RPM or 7,200 RPM drives. This will improve both read and write latency, but only by a factor of two or so. …

ZFS provides for the use of a separate intent-log device, a slog in ZFS jargon, to which synchronous writes can be quickly written and acknowledged to the client before the data is written to the storage pool. The slog is used only for small transactions while large transactions use the main storage pool – it’s tough to beat the raw throughput of large numbers of disks. The flash-based log device would be ideally suited for a ZFS slog. … Using such a device with ZFS in a test system, latencies measure in the range of 80-100µs which approaches the performance of NVRAM while having many other benefits. …

By combining the use of flash as an intent-log to reduce write latency with flash as a cache to reduce read latency, we can create a system that performs far better and consumes less power than other system of similar cost. It’s now possible to construct systems with a precise mix of write-optimized flash, flash for caching, DRAM, and cheap disks designed specifically to achieve the right balance of cost and performance for any given workload with data automatically handled by the appropriate level of the hierarchy. … Most generally, this new flash tier can be thought of as a radical form of hierarchical storage management (HSM) without the need for explicit management.

Updated July, 1: I’ve posted the link to the article in my subsequent blog post.

Back in January, I posted about a problem with Apple’s port of DTrace to Mac OS X. The heart of the issue is that their port would silently drop data such that certain experiments would be quietly invalid. Unfortunately, most reactions seized on a headline paraphrasing a line of the post — albeit with the critical negation omitted (the subject and language were, perhaps, too baroque to expect the press to read every excruciating word). The good news is that Apple has (quietly) fixed the problem in Mac OS X 10.5.3.

One issue was that timer based probes wouldn’t fire if certain applications were actively executing (e.g. iTunes). This was evident both by counting periodic probe firings, and by the absence of certain applications when profiling. Apple chose to solve this problem by allowing the probes to fire while denying any inspection of untraceable processes (and generating a verbose error in that case). This script which should count 1000 firings per virtual CPU gave sporadic results on earlier revisions of Mac OS X 10.5:

        @ = count();

On 10.5.3, the output is exactly what one would expect on a 2-core CPU (1,000 executions per core):

1  22697                         :tick-1s
1  22697                         :tick-1s

On previous revisions, profiling to see what applications were spending the most time on CPU would silently omit certain applications. Now, while we can’t actually peer into those apps, we can infer the presence of stealthy apps when we encounter an error:

        @[execname] = count();
        @["=stealth app="] = count();

Running this DTrace script will generate a lot of errors as we try to evaluate the execname variable for secret applications, but at the end we’ll end up with a table like this:

Adium                                                             1
GrowlHelperApp                                                    1
iCal                                                              1
kdcmond                                                           1
loginwindow                                                       1
Mail                                                              2
Activity Monito                                                   3
ntpd                                                              3
pmTool                                                            6
mlb-nexdef-auto                                                  12
Terminal                                                         14
=stealth app=                                                    29
WindowServer                                                     34
kernel_task                                                     307
Safari                                                          571

A big thank you to Apple for making progress on this issue; the situation is now much improved and considerably more palatable. That said, there are a couple of problems. The first is squarely the fault of team DTrace: we should probably have a mode where errors aren’t printed particularly if the script is already handling them explicitly using an ERROR probe as in the script above. For the Apple folks: I’d argue that revealing the name of otherwise untraceable processes is no more transparent than what Activity Monitor provides — could I have that please? Also, I’m not sure if this has always been true, but the ustack() action doesn’t seem to work from the profile action so simple profiling scripts like this one produce a bunch of errors and no output:

/execname == "Safari"/
        @[ustack()] = count();

But to reiterate: thank you thank you thank you, Steve, James, Tom, and the rest of the DTrace folks at Apple. It’s great to see these issues being addressed. The whole DTrace community appreciates it.

Recent Posts

February 12, 2017
December 18, 2016
August 9, 2016
August 2, 2016