Profile: Tucker

Profile: Andy Tucker

by James Turner

June 1, 2005

Andy Tucker has worked on the Solaris platform for practically his entire professional life, joining Sun immediately after completing graduate school at Stanford ten years ago. For most of that time, he's tinkered in one way or another with the deep dark internals of the Solaris kernel, finally becoming a distinguished engineer in the Solaris Kernel and Data Services group. Though Tucker has now moved on to become a Principal Engineer at VMware, he recently took some time to talk to OpenSolaris.org about the history and future of Solaris, as well as how a new technology called Solaris Zones may significantly change the way some people use Solaris servers.

OpenSolaris: How would you say Solaris has evolved during the time you were at Sun?

Tucker: I think when I first started at Sun, the most focus was really just on "Let's get this thing working," and it wasn't that things were falling over constantly, but it was very much a focus on stability, on improving performance. We had just switched over a couple years before from the SunOS 4.X, which is based on BSD code base, to what became Solaris, which was based on SVR4 from AT&T. So it was a completely different code base and there was a big perception around performance, particularly on smaller systems. This is when people started calling it "Slow-Laris" and that sort of thing. And so we had a big focus on both making it stable, making it work, getting the performance, improving performance; and also on figuring out what to do with the new hardware that Sun was starting to produce.

This was right around the same time that Sun was transitioning from what was essentially a workstation company to really having a server focus. At the time I started, they had recently come out with SPARC Center 2000, which was a 20-processor SPARC machine. A lot of what I was doing in addition to bug fixing and stability was figuring out how do we make things scale on systems of that size. And we were looking at the road maps, what eventually became the Ultra Enterprise 6000 and 10000, which were 32-processors, 64-processors, etcetera, and just trying to figure out how to scale across this kind of system while still being able to run on standard desktops and workstation environments.

So that was kind of where we were then. I think over the years, through the work that we've done, the focus has really changed and it's less to do with particular hardware or even stability, although obviously it needs to be important. But it's more on, "What can we do at the operating system level to provide new and interesting technology that gives value to customers?" So it's changing the focus from looking at the operating system as essentially a cost, where you want to minimize the amount of the system resources, to looking at what you can do at this level of the system in order to enable people who are using the system to think that they have the likelihood to be able to do things.

OpenSolaris: Obviously, the UNIX market has evolved tremendously over the ten years that you've been there, from being mainly proprietary to the free for all that it is today. What do you see Sun's role as now?

Tucker: As you said, there's a lot of interesting changes that have happened. I really think one of the roles that Sun has is that it's really the one company that has stood behind this idea of investing at the operating system level and continuing to evolve its version of UNIX toward the direction it seems to have, that the company considers valuable. I think if you look at, for example, everyone who is in the operating system business, I should say with the notable exception of Microsoft, who also invests at the operating system level and moves things in their own direction, but if you look at the other UNIX vendors, circa 1990-1993, they've all either adopted other operating systems technologies or are certainly showing a tendency to move in that direction. Less of a focus on the operating system technology that they, in a sense, own, and more of trying to move to one of the commodity platforms, either Windows or Linux. So that's really the distinguishing characteristic of where Sun is; I mean I really think this started in the mid-'90s when you had a lot of folks like SGI and HP and others who were starting to back away a little bit from their UNIX strategies and take a look at, "Should we go off and be doing Windows NT on our hardware platform?" and things like that. And it's certainly something that we, within Sun, looked at and considered, but in the end decided that wasn't workable --from a business point of view, from a technology point of view, and from what we're trying to bring to our customers.

OpenSolaris: Now, getting to your particular piece of specialty, you worked heavily on zones.

Tucker: Zones provide a way to encapsulate or isolate applications, multiple applications in the same system, in such a way that to each application it looks like it has a system to itself. And that's kind of the bottom line there. Really, the idea is to be able to allow you to improve your hardware utilization, system utilization, by being able to consolidate more things onto the same system. And I mean, we've had different technologies in the past to be able to support this, particularly the CPU scheduler and other resource management capabilities, but the big problem has always been that many applications aren't designed to be run in an environment where other things are in the same space. An example might be a web server that grabs port 80 on all available interfaces and doesn't have configuration capability in order to change that. Or maybe the person installing and administering it doesn't understand how to configure it in order to limit the addresses it binds to. So with zones, I can run 10, 20, 30 of those web servers, each in their own container, their own Zone, where they have some dedicated IP address for that specific application and some part of the filesystem and other system resources that are dedicated to that application, but that application doesn't see other applications, it doesn't interfere with their data or their activity; it's completely isolated from them. This can extend to users, too; you can give somebody a user login where they can log into a Zone on the system. They're able to see other users logged into that same Zone; they're able to do things with files and other kinds of applications within that Zone; but they're not able to affect the rest of the system.

OpenSolaris: I can already see a couple of differences between this and using chroot and quotas, but I could, as devil's advocate, say, "Well, you know, I could just use chroot and quotas and get a lot of that functionality." How is zones fundamentally different from them?

Tucker: You can get some of that functionality with chroot and other similar techniques, and particularly the ability to isolate the filesystem component. I think the big difference we did with zones is try to do a top-to-bottom complete application environment or as complete as possible. And so that means that not only, as in chroot, do you have only a part of a filesystem accessible to applications running within the Zone, but we also filter or isolate the process space, say what processes are visible. We isolate the IPC mechanisms that many applications use; System 5 IPCs, semaphores, shared memory, etcetera; so each one has its own shared memory space that's completely separated from other applications. We isolate the network interfaces; so if I just use chroot and quotas and run an application, it actually has access to all the network interfaces that are attached to that system. There's no way to restrict that. We built things like a virtual console so I can monitor what's going on with the console messages, especially with a particular Zone. We virtualize the system log daemon, so when applications run and they log messages to the system, it actually shows up somewhere that's visible to an administrator running within the Zone.

OpenSolaris: Now you can have more zones than you have CPUs?

Tucker: Yeah; it's all implemented in the operating system level. So you can either dedicate particular CPUs to particular zones; so I can say, "This Zone runs on this set of four CPUs," if you have a large system and relatively low number of zones, or you can divide them up according to shares of the CPU, where I can say, "This Zone gets five shares and that Zone gets ten shares and that Zone gets twenty shares." The one with twenty shares gets twice as much CPU as the one with ten shares. You can use a combination of those; but you can also run without any constraints. You can run them as you would on a normal system, and if the amount of CPUs that each one gets isn't particularly interesting or important, that might be a reasonable thing to do.

OpenSolaris: If I were to take application X, you know, some benchmarking application, and run it as root on a bare system and then run it in a Zone, would I see a performance hit?

Tucker: No, we haven't measured any difference in performance there. A virtual machine works by building a software layer that emulates the complete virtual machine that looks exactly like a physical machine to the operating system running within it. In our case, we didn't really need to do that; all we need is an application environment layer that is isolated from other things. Now, the interesting thing there is that the operating system is already managing the application environment and interactions between processes and interactions between processes and system resources. So, for example, that's what limits my ability from one process to go and send a signal or otherwise affect activity of another process; the operating system knows whether they're authorized to do that. So what we did is we went through and in the same exact places where we're doing those sorts of checks of inter-process communication and activity, we can insert checks that also ask, "Are you in the same Zone?" If you're in the same Zone, then we do the normal authorization checks. If you're in different zones, you can't do this at all; in fact, you can't even see that the other process exists.

OpenSolaris: I assume that someone running as root on, kind of the base system, sees everything?

Tucker: If you're running a root on the base system, you can see everything and control everything. We call that the global Zone, which is the standard environment that you're used to on Solaris on UNIX systems where as root you can see anything and do everything. So you can also run as root inside one of these non-global zones, these individual application zones, and you could do the normal privileged activities that are isolated to that Zone, like being able to create setuid applications, or being able to bind to privileged ports, but you won't be able to do things that will affect other zones for the system as a whole.

OpenSolaris: Like running /etc/shutdown.

Tucker: Well you can shut down your Zone, you can type reboot or shutdown and it will shut down your Zone, but you can't shut down the system as a whole; it's just localized to that specific Zone. You can't load random kernel modules because that could be a security violation.

OpenSolaris: The obvious place where I think that this has a big footprint immediately is the virtualized server market.

Tucker: This kind of technology has been used in the past, particularly used with the jail facility in FreeBSD, and particularly in the virtual web server type market, as you mentioned. I think it's been used fairly successfully there. Our main focus with this was to take that same basic idea and kind of complete the functionality and create a complete package in such a way that we can actually sell this into data-center environments where we can go have somebody who is administering databases and things like that, and they can very easily understand how they can consolidate their applications. And even to the point of being able to deal with operational and cultural issues within their data centers, like the fact that commonly there's an administrator associated with each application and then there's an administrator who runs the system as a whole, and so when doing consolidation you need to make sure that the administrators associated with each of the applications can't step on each other.

OpenSolaris: Now, looking at the open sourcing of Solaris, which is some value of imminent, what are the technological benefits that it could bring to Solaris, and what kind of community do you see evolving out of that?

Tucker: I think the technological value that it's going to bring into play is really the ability to take the existing system that we have and create the dialogue with our users or customers, developers, people who want to participate in the Solaris community around the components of the system. To bring in people who are interested in helping to contribute, bring in people who are interested maybe in just giving us feedback in a more direct fashion than they otherwise have been able to, and try to create a conversation, if you will, among people who are interested in improving and continuing this technology base. And some of those people might be Sun customers; some of them might not be, but really putting the focus on the Solaris technology that we've evolved over the years and bringing more people into the discussion about how should it be evolved over the next ten years. Where should we take this thing, what are the interesting problems that we need to solve next, and trying to enlist input and help with moving in those next directions. At the same time, preserving the basic values that we have and that we've used in terms of how we've created it and evolved it in the past with a strong focus on stability, quality, and compatibility.

OpenSolaris: One area that's been discussed a lot as an area that outside developers might be able to bring to the table is device drivers. What do you think are the areas that are kind of like the low-hanging fruit in terms of things that outside developers could bring to the table?

Tucker: Yeah, device drivers are an obvious example. I think that there's a lot of places where we have maybe some basic infrastructure, but we could certainly use tools that take advantage of that infrastructure in order to improve the usability of the system. There's a large amount of potential in that area, command-line tools, GUIs, etcetera; being able to help tie together Solaris with other open source projects.

OpenSolaris:Could you see something like ext3fs support or something like that making its way into the Solaris kernel?

Tucker:Yeah; that's certainly possible, creating sort of compatibility layers and things like that. We'd be able to mount that same filesystem across multiple systems and that sort of thing. Another area within Sun where we'd be interested in a community perspective is tying in together with some of the open source management tools like Webmin and being able to improve the administer-ability and usability on that basis. I think there's frankly a lot of work that needs to be done on Solaris in the modern desktop usability space. Things like multi-media support, support for some of the newer devices; a lot of that we're working on, but we can certainly use help.

OpenSolaris: What do you recommend for someone starting fresh on the OpenSolaris code?

Tucker:The first thing to do is to join the mail aliases and start participating in the discussions. You should also read the archives to get a sense of what's been talked about before. Beyond that, it depends on how you want to participate. If you want to write code, I suggest starting with some simple bug fixes or enhancements. We're planning to list some of these on the website, to make it easy to find a "first bite" type project. But it's not just about the code--you can help out by testing the bits, contributing to FAQs and other project documentation, filing bugs or feature requests, contributing to design discussions, or even just talking with the rest of the community about how we can improve things.

 James Turner is the president of Black Bear Software and a freelance journalist.