OpenSolaris Observability
Over the course of history, we've done a lot of debugging in Solaris. Whether it's processes misbehaving, performance degradations, system panics, or hardware failure, there is one key element of any approach:
If you cannot observe the problem, you cannot fix it.
To this end, we have spent a great deal of time on robust tools to observe all aspects of system behavior. Some of these have been around forever; others are relatively new. They all make the lives of administrators and developers much easier. This community will serve to explore existing Solaris tools and features, as well as plan the next generation of OpenSolaris observability tools.
We've divided this rather large cross section of Solaris functionality into several specific areas, described here. Most of these just link to the relevant code. Over time, those that are sufficiently complex will link to more detailed documentation.
Process Observability
| DTrace | A fantastic tool that can observe process and system behavior and tie it all together. Check out the DTrace community for more information. |
| truss | A tool for examining system calls as well library calls made by an application. |
| ptools | A suite of process observability tools. |
| libproc | A common library used to simplify examining and manipulating processes. |
| procfs | The /proc filesystem, upon which libproc and all process tools are built. |
| MDB | The modular debugger, the standard debugger for Solaris. See the MDB community for more information |
| ps | Basic process monitoring tool. Displays a snapshot of current process state on the machine. |
| prstat | The standard process monitoring tool for Solaris, similar to top. |
| plockstat | The userland equiavlent of lockstat(1M), used to monitor lock events and aid in MT scalability. Built on top of DTrace |
System Observability
| DTrace | Once again, the indispensible system monitoring tool. |
| kstat | A kernel statistics framework used by a variety of tools to examine system wide statistics. |
| NUMA | Tools for examining NUMA (Non Uniform Memory Architecture) characteristics on Solaris. |
| ctfs | The contract filesystem, used to observe and manipulate process contracts. |
| objfs | The object filesystem, used to examine kernel module symbols and CTF data from userland. |
| mpstat iostat vmstat | Tools to report on basic system statistics. |
| lockstat | Tools to examine kernel locking statistic, as well as basic profiling data. Implemented using DTrace |
| intrstat | Tools to examine kernel interrupt statistics. Implemented using DTrace |
| trapstat | SPARC-only tool to examine trap statistics. |
Network Observability
| snoop | The standard tool for observing network packets. |
| netstat | Tool for observing network connection status. |
| nfsstat | Basic tool for monitoring NFS statistics. |
Hardware Observability
| FMA | Fault Management Architecture, a unified system of hardware fault diagnosis, repair, and reporting. |
| CPC | A system to gather and report CPU performance counter information. Consists of kernel components, libcpc, cpustat, and cputrack |
| psrinfo | Report on current processor status and configuration |
| prtconf | Display the system device tree, optionally in excrutiating detail. |
| prtpicl | Similar to prtconf, prints out the state of the device nodes maintained by the PICL daemon. |
| prtdiag | Display summary of attached hardware components. |
| prtfru | SPARC only - Display FRUID information for system or domain. |
Post Mortem Observability
| MDB | The standard debugger. See the MDB community for more information. |
| dumpadm | Configures kernel crash dump generation. |
| coreadm | Configures process core file generation. |
| CTF | Compact C Type Format, the format used to store type information in the kernel and userland. Consumed by MDB and DTrace, among others. |
| process core generation | The elfcore() routine, which is responsible for the nuts and bolts of core file generation. |
| kernel panic code | The entry point to the kernel panic code. |
| gcore | The gcore(1) utility for taking a core dump of a live running process without perturbing the process. |
Unbundled Tools
More to come...