Adam Leventhal's blog

Close this search box.

Month: May 2011

Apple recently announced a new iMac model — in itself, only as notable as the seasons — but with an interesting option: users can choose to have both an HDD and an SSD. Their use of these two is absolutely pedestrian, as noted on the Apple store:

If you configure your iMac with both the solid-state drive and a Serial ATA hard drive, it will come preformatted with Mac OS X and all your applications on the solid-state drive. Then you can use the hard drive for videos, photos, and other files.

Fantastic. This is hierarchical storage management (HSM) as it was conceived by Alan Turing himself as he toiled against the German war machine (if I remember my history correctly). The onus of choosing the right medium for given data is completely on the user. I guess the user who forks over $500 for this fancy storage probably is savvy enough to copy files to and fro, but aren’t computers pretty good at automating stuff like that?

Back at Sun, we built the ZFS Hybrid Storage Pool (HSP), a system that combines disk, DRAM, and, yes, flash. A significant part of this is the L2ARC implemented by Brendan Gregg that uses SSDs as a cache for often-used data. Hey Apple, does this sound useful?

Curiously, the new iMacs contain the Intel Z68 chipset which provides support for SSD caching. Similar to what ZFS does in software, the Intel chipset stores a subset of the data from the HDD on the SSD. By the time the hardware sees the data, it’s stripped of all semantic meaning — it’s just offsets and sizes. In ZFS, however, the L2ARC knows more about the data so can do a better job about retaining data that’s relevant. But iMac users suffer a more fundamental problem: the SSD caching feature of the Z68 doesn’t appear to be used.

It’s a shame that Apple abandoned the port of ZFS they had completed ostensibly due to “licensing issues” (DTrace in Mac OS X uses the same license — perhaps a subject for another blog post). Fortunately, Ten’s Complement has picked up the reins. Apple systems with HDDs and SSDs could be the ideal use case ZFS in the consumer environment.

It’s rare to get software right the first time. I’m not referring to bugs in implementation requiring narrow fixes, but rather places in a design that simply missed the mark. Even if getting it absolutely right the first time were possible, it would be prohibitively expensive (and time-consuming) so we make the best decisions we can, hammer it out, and then wait. Users of a product quickly become experts on its strengths and weaknesses for them.  Customers aren’t beta testers — again, I’m not talking about bugs — but rather they expose use cases you never anticipated, and present environments too convoluted to ever conceive at a whiteboard.

When I worked in the Fishworks group at Sun, we learned more about our market in the first three months after shipping 1.0 than we had in the 30 months we spent developing it. We found the product both struggling in unanticipated conditions, and being used to solve problems we could have never predicted.  Some of these we might have guessed earlier given more time, but some will never come to light until you ship. That you need to ship 1.0 before you can write 2.0 is a deeper notion than it appears.

I joined Delphix a couple of weeks before our formal launch at the DEMO conference. Since then, we’ve engaged in more proofs-of-contept (PoCs) and more customers have rolled us into use, and we’ve continued to learn of new use cases for Delphix Server, and found the places where we needed to rethink our assumptions. And we knew this would be the case — you can’t get it right the first time. Over the past several months, we in Delphix engineering have been writing the second version of the most critical components in our stack, incorporating the lessons learned with our customers. The team has enjoyed the opportunity to revisit design decisions with new information; it’s fun to feel like we’re not just getting it done, but getting it right.

When building 1.0, you make a mental list of the stuff you’d fix if only you had the time. In the next release, you figure out all the whizzy stuff you now can build on the stable foundation. We’re excited for the forthcoming 2.6 release — more so even for new ideas we found along the way that will be the basis for our future work. We’ve got a great team working on a great product. Check in on the Delphix blogs in the coming months for details on the 2.6 release and the other stuff we’ve got in the works.

Recent Posts

January 22, 2024
January 13, 2024
December 29, 2023
February 12, 2017
December 18, 2016
August 9, 2016