Category: OpenSolaris

While trawling through b.s.c., a comment caught my eye in this post from Glenn’s weblog:

As a shareholder, I do NOT want you to “open source” solaris in its entirety (ESPECIALLY DTrace!). I want you to keep the good stuff completely sun-only, accessible only under NDA.

Certainly, this echoes some of the same concern I had when I started hearing rumblings about OpenSolaris — we in Solaris have spent years of our lives making these innovations (ESPECIALLY DTrace!), and we don’t want to see them robbed. I’m also a shareholder and I don’t want to see my investments of time, effort, and — forgive me — money go to waste.

Now that I know more about OpenSolaris and open source in general, I’m confident that Sun isn’t selling out Solaris or giving away the company’s crown jewel, rather we’re going to make Solaris better, and more widely used. That sounds a little Rah Rah Solaris, but let’s look more closely at OpenSolaris and what it might mean for Solaris and for Sun (and for the author of the comment, a shareholder).

Open source is an interesting dichotomy: on one side there are the developers and the community with the spirit of the free trade of software and ideas, and on the other side there are the Linux vendors selling service contracts to fat cat customers. The former is clearly the benefit of OpenSolaris — a larger community of developers and users will improve Solaris and grow its audicent. The latter is the potential risk — we’re concerned that other companies might directly steals Sun’s customers by using Sun’s technology. The specifics of the OpenSolaris license haven’t been finalized so it’s possible that the license and patents will prevent Linux vendors from selling technologies developed in Solaris outright. Regardless, Solaris isn’t just a bunch of code, it’s the support and service and documentation and us, the Solaris developers.

When a Sun customer pays for Solaris, they’re paying for someone over here to answer the phone when they call and for me and others in Solaris kernel development to do the things they need. Even when they source code is available, customers will still want to tap into the origins of that code and talk to the people who made it. If there are problems they’ll want to be able to rely on the experts to fix them.

What about documentation? The Solaris Dynamic Tracing Guide is still going to be free but only as in beer — we’re not open sourcing our documentation (as least as far as I know). So let’s say dtrace.c was dropped into Linux, would they then rewrite the entire answer book (400 pages and counting) from scratch? Maybe this wouldn’t matter much to ordinary users, but if you’re giving some Linux vendor a big sweaty wad of cash to support DTrace on Linux you expect some documentation! The Solaris docs would be close enough for some users, but not customers shelling out the big big dollars for a service contract.

Even if Linux were able to replicate DTrace and document it and a linux vendor were able to support it, I’m confident the existing and growing Solaris community could keep innovating and push Solaris ahead. On a more person note, I’m also excited about OpenSolaris because it means if I were ever to leave Sun, I could still work on DTrace, mdb, nohup, and the other parts of Solaris that I consider my own.

OpenSolaris can only help Sun. If it succeeds, there will be a larger community of Solaris developers making it work with more platforms and devices, fixing more problems, and improving the quality of life on Solaris which will spawn an even larger community of Solaris users, both individuals and paying customers; if OpenSolaris fails, then that it won’t help to create those communities, and I think that’s the only consequence.

go to the Solaris 10 top 11-20 list for more

p-tools

Since Solaris 7 we’ve included a bunch of process observability tools — the so called “p-tools”. Some of them inspect aspects of the process of the whole. For example, the pmap(1) command shows you information about a process’s mappings, their location and ancillary information (the associated file, shmid, etc.). pldd(1) is another example; it shows which shared objects a process has opened.

Other p-tools apply to the threads in a process. The pstack(1) utility shows the call stacks for each thread in a process. New in Solaris 10 Eric and Andrei have modified the p-tools that apply to threads so that you can specify the threads you’re interested in rather than having to sift through all of them.

pstack(1)

Developers and administrators often use pstack(1) to see what a process is doing and if it’s making progress. You’ll often turn to pstack(1) after prstat(1) or top(1) shows a process consuming a bunch of CPU time — what’s that guy up to. Complex processes can many many threads; fortunately prstat(1)’s -L flag will split out each thread in a process as its own row so you can quickly see that thread 5, say, is the one that’s hammering the processor. Now rather than sifting through all 100 threads to find thread 5, you can just to this:

$ pstack 107/5
100225: /usr/sbin/nscd
-----------------  lwp# 5 / thread# 5  --------------------
c2a0314c nanosleep (c25edfb0, c25edfb8)
08056a96 gethost_revalidate (0) + 4b
c2a02d10 _thr_setup (c2949000) + 50
c2a02ed0 _lwp_start (c2949000, 0, 0, c25edff8, c2a02ed0, c2949000)

Alternatively, you can specify a range of threads (5-7 or 11-), and combinations of ranges (5-7,11-). Giving us something like this:

$ pstack 107/5-7,11-
100225: /usr/sbin/nscd
-----------------  lwp# 5 / thread# 5  --------------------
c2a0314c nanosleep (c25edfb0, c25edfb8)
08056a96 gethost_revalidate (0) + 4b
c2a02d10 _thr_setup (c2949000) + 50
c2a02ed0 _lwp_start (c2949000, 0, 0, c25edff8, c2a02ed0, c2949000)
-----------------  lwp# 6 / thread# 6  --------------------
c2a0314c nanosleep (c24edfb0, c24edfb8)
080577d6 getnode_revalidate (0) + 4b
c2a02d10 _thr_setup (c2949400) + 50
c2a02ed0 _lwp_start (c2949400, 0, 0, c24edff8, c2a02ed0, c2949400)
-----------------  lwp# 7 / thread# 7  --------------------
c2a0314c nanosleep (c23edfb0, c23edfb8)
08055f56 getgr_revalidate (0) + 4b
c2a02d10 _thr_setup (c2949800) + 50
c2a02ed0 _lwp_start (c2949800, 0, 0, c23edff8, c2a02ed0, c2949800)
-----------------  lwp# 11 / thread# 11  --------------------
c2a0314c nanosleep (c1fcdf60, c1fcdf68)
0805887d reap_hash (80ca918, 8081140, 807f2f8, 259) + ed
0805292a nsc_reaper (807f92c, 80ca918, 8081140, 807f2f8, c1fcdfec, c2a02d10) + 6d
08055ded getpw_uid_reaper (0) + 1d
c2a02d10 _thr_setup (c20d0800) + 50
c2a02ed0 _lwp_start (c20d0800, 0, 0, c1fcdff8, c2a02ed0, c20d0800)
...

The thread specification syntax also works for core files if you’re just trying to drill down on, say, the thread that caused the fatal problem:

$ pstack core/2
core 'core/2' of 100225:        /usr/sbin/nscd
-----------------  lwp# 2 / thread# 2  --------------------
c2a04888 door     (c28fbdc0, 74, 0, 0, c28fde00, 4)
080540bd ???????? (deadbeee, c28fddec, 11, 0, 0, 8053d33)
c2a0491c _door_return () + bc

truss(1)

The truss(1) utility is the mother of all p-tools. It lets you trace a process’s system calls, faults, and signals as well as user-land function calls. In addition to consuming pretty much every lower- and upper-case command line option, truss(1) now also supports the thread specification syntax. Now you can follow just the threads that are doing something interesting:

truss -p 107/5
openat(-3041965, ".", O_RDONLY|O_NDELAY|O_LARGEFILE) = 3
fcntl(3, F_SETFD, 0x00000001)                   = 0
fstat64(3, 0x08047800)                          = 0
getdents64(3, 0xC2ABE000, 8192)                 = 8184
brk(0x080721C8)                                 = 0
...

pbind(1)

The pbind(1) utility isn’t an observability tool, rather this p-tool binds a process to a particular CPU so that it will only run on that CPU (except in some unusual circumstances; see the man page for details). For multi-threaded processes, the process is clearly not the right granularity for this kind of activity — you want to be able to bind this thread to that CPU, and those threads to some other CPU. In Solaris 10, that’s a snap:

$ pbind -b 1 107/2
lwp id 107/2: was not bound, now 1
$ pbind -b 0 107/2-5
lwp id 107/2: was 1, now 0
lwp id 107/3: was not bound, now 0
lwp id 107/4: was not bound, now 0
lwp id 107/5: was not bound, now 0

These are perfect examples of Solaris responding to requests from users: there was no easy way to solve these problems, and that was causing our users pain, so we fixed it. After the BOF at OSCON, a Solaris user had a laundry lists of problems and requests, and was skeptical about our interest in fixing them, but I convinced him that we do care, but we need to hear about them. So let’s hear about your gripes and wish lists for Solaris. Many of the usability features (the p-tools for example) came out of our own use of Solaris in kernel development — once OpenSolaris lets everyone be a Solaris kernel developer, I’m sure we’ll be stumbling onto many more quality of life tools like pstack(1), truss(1), and pbind(1).

This past week at OSCON I’ve spent my time trying to understand open source processes, talking about Solaris, and trying to figure out what OpenSolaris is going to look like.

Learning from Linux

I attended a talk by Greg Kroah-Hartman about Linux kernel development. As we work towards open sourcing Solaris, we’re trying to figure out how to do it right — source control, process, licenses, community etc. As I didn’t know much about how Linux development works, I was hoping to learn from a largely successful open source operating system.

Linux development is built around fiefdoms maintained by folks like Greg. Ordinary folks can contribute to the repositories they maintain (either directly or by proxy based on some sort of Linux-street-cred it seems). Those repositories are then fed up to a combined unstable repository, and from there Linus himself ordains the patches and welcomes them into the circle of linux 2.6.x. This all seemed to make some sense and work alright. That is, until someone asked about firewire support. The answer, “I wouldn’t run firewire — it should build [laughter], but I wouldn’t run it. The discussion then led to Linux testing which it seems is highly ad-hoc and unreliable. IBM and Novell are working on nightly testing runs, but very little exists today in terms of quality control tests or general tests that developers themselves can run before they integrate their changes.

In Solaris, testing can be arduous. Some changes are obvious and can be tested on just on architecture, but others require extensive tests on a variety of SPARC and x86 platforms. And linux supports so many more platforms! I have no idea how a developer working on his x86 box can ever be sure that some seemingly innocuous change hasn’t broken 64-bit PPC (or whatever). Clearly this is something we have to solve for OpenSolaris — reliability and testing are at the core of our DNA in the Solaris kernel group, and we need to not only export that idea to the community, but only some subset of facilities so that contributors can adhere to the same levels of quality.

OpenSolaris

Later in the day a bunch of us from Solaris met with some open source leaders (I don’t know quite how one earns that title, but that’s what our liaison told us they were). We first told them where we were: Yes, we really are going to open source Solaris; no, we don’t know the license yet; no, we don’t know if it’s going to be GPL compatible; no, we aren’t planning on moving the cool stuff in Solaris over to Linux ourselves; and, no, we do not know what the license is going to be, but we promise to tell you when we do.

We got a lot of helpful suggestions from folks involved with apache and other projects. “Bite sized bugs” sound like a great way to get new people involved with Solaris and contributing code without a huge investment of effort. Documentation, partitioning and documenting that partitioning will all be much more important that we had previously anticipated. We get the message: OpenSolaris will be easy to download, build, and install, and we’ll make sure it’s as easy as possible to get started with development.

The one thing that disappointed me was the lack of knowledge about Solaris 10 — some comment suggested that the members of the panel think Solaris doesn’t really have anything interesting. Fortunately we had the BOF that night…

The Solaris Community

Andy, Bart, Eric and I held a BOF session last night to talk about OpenSolaris and Solaris 10. After satiating the crowd’s curiosity about open sourcing Solaris (no, we don’t know what the license is going to be), we gave some Solaris 10 demonstrations.

Since we were tight on time, I buzzed through the DTrace demo in about 15 minutes touching on the syscall provider, aggregations, the ustack() action, user-land tracing with the pid provider and kernel tracing with the fbt provider. Whew. Then came the questions — some of the audience members had used DTrace, others had heard of it; almost everyone had a question. When I demo DTrace, there’s always this great moment of epiphany that people go through. I can see it on their faces. After the initial demo they look like people who just got off a roller coaster — windblown and trying to understand what just happened, but there’s always this moment, this ah-ha moment, when something — the answer to a question, an additional example, an anecdote — sparks them into understanding. It’s great to see someone suddenly sit up in her chair and start nodding vigorously at every new site I point out in the DTrace guided tour.

My personal favorite piece of input about OpenSolaris was someone’s claim that six months after Solaris goes open source there will be a port to PowerBook hardware. If that’s true then everyone in the Solaris kernel group is going to have PowerBooks in 6 months plus a day.

Before the BOF, I was worried that we might not find our community for open source Solaris. Not only was there a good crowd at the BOF, but they were interested and impressed. That’s our community, and those are the people who are going to be contributing to OpenSolaris. And I can’t wait for it to happen.

Two members of the ZFS team have joined the blogging fray. Check out Matt Ahrens‘s and Val Henson‘s weblogs.

For the unintiated, ZFS is the brand new file system that’s going to be in Solaris 10. ZFS is incredibly fast, reliable, and easy to manage. I recently moved my home directory from out UFS file server to an experimental ZFS file server. Opening my (extensive) mail spool went from 20 seconds to 3; doing an ls(1) sped up by more than a factor of two in my home directory; and the repository of crash dumps I keep went from 40G to 4G. This is really cool technology both under the hood and from the point of view of users and administrators — stay tuned to their weblogs for all the details.

This afternoon, I’m leaving for OSCON (easily confused with the bi-mon-sci-fi-con). Here in Solaris Kernel Development we’ve been talking a bunch about the impending open sourcing of Solaris and what that’s going to look like. I’m very excited about OpenSolaris itself, and I’m looking forward to talking to folks at OSCON to hear what they think.

The part they’re going to find most surprising is that this is for real. Within a year or two, there are going to be people from outside of Sun contributing to Solaris. Period. It’s going to be a little scary, but I’m excited about the possibilities of what this might mean for Solaris (my dream of Solaris on my PowerBook might even come true).

We’re holding a BOF session on Thursday at 9pm. So come by if you want to talk to Andy, Eric, Bart or me about the cool stuff in Solaris 10 or about open sourcing Solaris. Stay for the free (as in beer) beer.

Another linker alien has joing the b.s.c. fray. Mike Walker already has some useful stuff about shared libraries that you should check out. If you have linker questions, do what I do: ask Mike.