Zone Statistics

Overview and Goals

 Zone Statistics is an OpenSolaris project devoted to developing a better understanding of resource consumption and resource controls of Solaris Containers (aka Solaris Zones) and their workloads. The goal of the project is a software tool which reports on resource consumption and, optionally, compares such consumption to resource controls which have been set. A prototype exists, written in Perl. Output is shown in the section "History and Description.'

Introduction

 Studies have shown that the average CPU utilization of data center computers is roughly 10-15%. The rest of the CPU capacity of the system is wasted. This has led to a desire to consolidate workloads on under-utilized systems. On some operating systems, this is difficult. Solaris has demonstrated its strengths as a consolidation platform for many years, but there are situations that call for strong isolation between consolidated workloads.

 Solaris Containers provide that isolation. A factor in workload isolation is resource management &mdash the ability to balance the use of system resources among competing workloads, as directed by the platform administrator. This can include prioritization of workloads, exclusive assignment of one workload to a CPU or group of CPUs, limitations on total RAM usage by a workload, and other resource controls.

 Unfortunately, monitoring resource usage and controls requires the use of many commands or a heavyweight GUI. There is a need for a single-command 'dashboard' that displays each Container's use of system resources, optionally providing a comparison to the current resource controls set for each Container. This tool should be analogous to the Solaris *stat tools (vmstat, mpstat, et al.)

Download

 There is a prototype, written in Perl. You can download the Perl script in the 'Files' section of this project. There is a link to 'Files' in this page's navigation bar.

History and Description

Discuss!

 Discussions for Zone Statistics should be directed to zones-discuss -at- opensolaris.org .

Problems

CPU Hog

Zonestat's biggest problem is due to its brute-force nature. It runs a few commands for each zone that is running. This can consume many CPU cycles, and can take a few seconds to run with many zones.

Wrong / Misleading CPU Usage Data

 Two commonly used methods to 'measure' CPU usage by processes and zones are prstat and mpstat. Each can produce wildly inaccurate 'data' in certain situations.

 With mpstat, it is very easy to create surprising results. For example, on a CMT system, set a CPU cap on a zone in a pool, and run a few CPU-bound processes. Run mpstat in the zone, and it will underestimate the amount of CPU used. This is due to the method used by mpstat to calculate its data.

 The prstat command only computes its data occasionally, ignoring anything that happened between samples. This command undercounts CPU usage for zones that have many short-lived processes.

 I wrote code to gather data from each, but prstat seemed more useful, so the output comes from prstat.

What's Next

 I would like feedback on this tool, perhaps leading to minor modifications to improve its robustness and usability. What's missing? What's not clear?

The future of zonestat might include these:

• I hope that this can be re-written in C or D. Either way, it might find its way into Solaris... If I can find the time, I would like to tackle this, but would need some assistance.
• New features - best added to a 'real' version:
  1. -d: show disk usage
  2. -n: show network usage
  3. -i: also show installed zones that are not running
  4. -c: also show configured zones that are not installed
  5. -p: only show processor info, but add more fields, e.g. prstat's instantaneous CPU%, micro-state fields, and mpstat's CPU%
  6. -m: only show memory-related info, and add the paging columns of vmstat, output of "vmstat -p", free swap space
  7. -s : sort sample output by a field. Good example: sort by poolID
  8. Deal with prstat vs. mpstat, perhaps by using DTrace
  9. Add a one-character column showing the default scheduler for each pool
  10. Report state transitions like mpstat does, e.g. changes in zone state, changes in pool configuration
  11. Improve robustness
  12. Improve the 'shorten' routines to increase precision if it will fit, e.g. convert 1234M to 1.2G instead of 1G