Fault Management Community
About Predictive Self-Healing
Self-healing functionality for users and administrators of a modern operating system provides fine-grained fault isolation and restart where possible of any component — hardware or software — that experiences a problem. To do so, the system must include intelligent, automated, proactive diagnoses of errors that are observed on the system. The diagnosis system is used to trigger targeted automated responses or guided human intervention that mitigates a specific problem or at least prevents it from getting worse. Finally, these new system capabilities are connected to a new model for system administrators oriented around simpler, higher-level abstractions. Sun's first Predictive Self-Healing features are part of the OpenSolaris OS and the Solaris 10 OS and include the Fault Manager and the Service Manager.
About Fault Management
The Solaris Fault Management effort (originally code-named FMA inside of Sun) provides a new architecture for building resilient error handlers, error telemetry, automated diagnosis software, response agents, and a consistent model of system failures for a management stack. Many parts of Solaris are already participating in FMA, including the CPU and Memory error handling for UltraSPARC III and IV, the UltraSPARC PCI HBAs, and more. And a variety of projects are underway, including full support for CPU, Memory, and I/O faults on Opteron, conversion of key device drivers, and integration with various management stacks.
The legacy UNIX failure model was simply to leave error handling up to each subsystem author, and simply provide the ability to emit an error message for a human to the system log in a non-standard format. When a subsystem is converted to participate in Fault Management, error handling is made resilient so that the system can continue to operate despite some underlying failure, and telemetry events are produced that drive automated diagnosis and response. The Fault Management tools and architecture enable development of self-healing content for software and hardware failures, for both microscopic and macroscopic system resources, all with a unified, simple view for administrators and system management software.
Some objectives for the Fault Management Community are to:
- Convert hardware and software subsystems to participate in Fault Management
- Connect Solaris Fault Management to system management protocols and software
- Evolve and enrich the tools and common architecture for Fault Management
- Research underlying failure modes and design and develop automated diagnosis software that is able to build effective self-healing for particular resources
Information Resources
Discussion Forum
Participate in Fault Management discussions.
fm-discuss General discussion of the tools, technology, infrastructure, and content to permit Solaris and Sun products to recover from, diagnose, isolate, and explain faults
Jive forum interface or Mailman interface
To subscribe, see "Subscribing to fm-discuss" on the mailman page or send an empty email to fm-discuss dash subscribe at opensolaris dot org
Documentation
- FMA Events and Messages
- Diagnosis results obtained from the Fault Management software in Solaris contain links to the Knowledge Article Web.
- The FMA Event Registry is the central repository for all fault management events passed between error handlers, the fault manager and its agents.
- Fault Management for System Administrators. This Fault Management piece is pointed to from Working with the Fault Manager in the Troubleshooting section of the OpenSolaris System Administration Guide.
- Man Pages
- fmd(1M)
- fmadm(1M)
In OpenSolaris 2008.11, the fmadm repair command was replaced by the following new commands: fmadm repaired (synonymous with fmadm repair), fmadm replaced, and fmadm acquit. - fmdump(1M)
In OpenSolaris 2008.11, the preferred method to display fault information and determine the FRUs involved is the fmadm faulty command, not the fmdump command. - fmstat(1M)
- Writing Device Drivers for FMA. The Writing Device Drivers guide contains a section "Sun Fault Management Architecture I/O Fault Services" in "Chapter 13, Hardening Solaris Drivers." This section describes the steps and techniques used to write an FMA-aware driver.
- Fault Management Daemon Programmer's Reference Manual. The FMD PRM [PDF] is a description of the internal architecture of the Sun Fault Management Daemon, fmd(1M), and the programming interfaces exported by the daemon.
- FMDPRM 1.4 April 2008. Added -b option to the fmtopo command. Changed descriptions of TOPO_WALK_CHILD and TOPO_WALK_SIBLING.
- FMDPRM 1.3 March 2008. Added repaircode to the table of Fault Management Configuration Properties.
- FMDPRM 1.2 August 2007. Initial post of this document.
- Eversholt Fault Tree Description Language. The Eversholt Fault Tree Description Language [PDF] explains how to use the eversholt language to describe fault trees in the Sun Fault Management Architecture.
- Eversholt Fault Tree Description Language, 1.8, November 2008. Initial post of this document.
What's New
The OpenSolaris universe of new fault management activities (projects, ARC cases, diagnosis engines, recovery agents and error handlers).
