General Principles for Requiring Changes to Projects

General Principles for Requiring Changes to Projects

If a project is likely to succeed as proposed, but could clearly be measurably improved by additional changes, the ARC should recommend those changes as ``Technical Changes Advised'' (or TCAs).

If a project is seriously flawed (i.e. if we would almost surely regret having shipped it) the ARC should clearly spell out and require specific changes to correct those flaws as ``Technical Changes Required'' (or TCRs).

If a project is so seriously flawed that it is not possible to clearly enumerate a list of changes that would make it acceptable (or if the changes are so massive as to constitute a redefinition of the project) the committee should reject the case. Rejections are very rare, but when a case is rejected, it is vital that the opinion clearly explain the reasons why the case could not be approved.

In considering whether a change should or should not be required, the committee must exercise considerable judgement. The next section describes (with examples) types of problems that commonly result in TCRs to a project. The subsequent sections discuss situations where the rules should be tempered with reason, and where we have to determine whether or not a project goes far enough.

Problems that Commonly result in TCRs (or Rejections)

The key question is: ``what constitutes a serious flaw?'' What flaws are so serious that we would disapprove the project if they were not corrected? It is probably not possible to exhaustively enumerate all possible flaws that could be grounds for a TCR or rejection. Nonetheless, there is great value (for everyone involved) in trying to set reasonable expectations for what types of problems might or might not justify requiring changes to a project.

The following list is drawn from years of ARC experience, and probably covers most of the commonly encountered problems. It begins with relatively clear situations, and works its way up to more subtle and complex issues (where objective determinations may be harder to make). Most of these examples are drawn from actual cases, although some of the issues have been simplified for illustrative purposes.

1. The project, as proposed, violates official product requirements.

   example: a general desktop user tool that is not level 4 internationalized would violate Scott's I18N dictum.

   example: a new API that includes a data-structure (to be exchanged over RPC channels) with fields that changed length on 32- and 64-bit systems would violate 64-bit safety guidelines.

2. The proposed interfaces violate established conventions.
   example: an unbundled C++ compiler that replaces the standard version of /lib/libc.so.1 would violate the "prime directive" of packaging (multiple variants of a single component).

   example: a device hot-plug notification mechanism that defines new routines named "event_send" and "event_rcv" would be staking out inappropriately generic name-space.

3. The project is not sufficiently specified to permit a meaningful architectural review.
   example: a proposal that includes automatically generated Javadoc specifications for every method, but has no descriptions of the associated semantics.

   example: a proposal to add a DHCP server without describing how IP addresses are managed and how host-to-IP-address maps are updated.

4. The project, as proposed, is incomplete, or unlikely to satisfy even its own stated goals.
   example: a device driver interface that is supposed to enable third party developers to build and deliver unbundled device drivers without needing system sources, but that requires source level changes to a Sun-supplied platform support module in order to support a DMA device would not enable non-source customers to build device drivers.

   example: a high-availability fail-over mechanism that is obviously susceptible to dead-locking in normal operation would not deliver the advertised high availability.

5. The project does not satisfy reasonable expectations for functionality, robustness, security, interoperability, non-regression, or competitive features.
   example: an authentication system that specifies passwords with command-line options or transmits them (unencrypted) over a network would not satisfy reasonable security expectations.

   example: if the previous release automatically worked on platform X, but the new version will require manual configuration before it can be successfully booted on the same system, this would represent a functionality regression.

   example: Wintel users can eject a diskette at any time, and DOS format diskettes are expected to be used as interoperability media between Solaris and Wintel systems. The intended customers have a reasonable expectation that Solaris systems will not panic as a result of the ``unannounced removal'' of a diskette.

   example: a DOS FAT 16 file system is most likely to be used to read and write interchange media, and an implementation that cannot handle the extended file names written by Windows systems would surely upset many customers.

   example: if a DHCP server product is intended to be sold as an add-on to release X and also to be bundled with release X+1, it would be unreasonable to require end-users to completely reconfigure the server after they upgrade from release X to release X+1.

   example: three projects introduce meta-drivers that are to be pushed between the IP stack and the DLPI driver:

   • IP traffic encryption
   • network bandwidth management
   • network adaptor fail-over

1. The project, as proposed, is unlikely to be supportable for the intended customers.

   example: a third party packaging scheme for unbundled device drivers that requires all device drivers to have unique names, but provides no means by which suppliers could obtain, generate, or register unique names would create significant support problems.

2. The proposed architecture is unlikely to be able to grow to meet reasonably anticipatable future requirements.
   example: a cast-in-stone resource synchronization protocol that is likely to be used across an enterprise, but is limited to supporting 32 hosts, or whose cost increases as the square of the number of participating hosts.

3. The project imposes an undue burden on the consumer.
   example: an API that requires callers to regularly generate or retrieve a piece information that has no meaning to them, and that could be easily maintained within the implementation.

   example: a production API that requires numerous or expensive calls to perform performance-critical or common operations.

4. The project confuses customers with redundant interfaces.
   example: two different projects that need to configure different aspects of TCP/IP behavior each introduce their own configuration utilities (rather than merely updating the existing ifconfig utility).

5. The project introduces redundant mechanisms whose functionality does not justify the added support costs.
   example: a project to add multi-initiator shared SCSI support proposes to create a second generic SCSI driver, rather than enhancing the existing one.

   Implementation redundancy, however, may be more acceptable in third party products. Third party software is often developed to run on a wide range of operating systems, and using Sun-specific implementations might reduce its portability.

Tempering the Rules with Reason

In his oft quoted opinion on "Northern Securities vs United States", Oliver Wendell Holmes said:

Great cases, like hard cases, make bad law

In response to past mistakes, we formulate new rules. In most cases, the rules help us to avoid future fiascos. Occasionally, the rules prove to be simpler than the situations they govern. The Architectural Review Committees are responsible for interpreting and applying the rules in the context of each individual case - and so they must be prepared to make exceptions in situations where the rules don't make sense.

This process may be the most subtle and subjective part of an ARC member's job. Unfortunately, we cannot suggest any rules to guide you through the process of bending the rules. We can, however, describe a few situations where committees have decided to grant exceptions:

example: 100% compliance may not be the right thing.

A project proposed to exchange textural messages through a standard over-the-wire protocol that had no provisions for internationalization. Such an implementation would seem to violate the level 4 internationalization mandate.

The project did fully internationalize all of its own messages and user dialogs, but asserted that changing the protocol to accommodate variable text and character sets would result in an application that would not interoperate with other clients. The committee agreed that the team had complied with the rule in all reasonable respects, and that sacrificing interoperability to gain internationalization of over-the-wire traffic would be a bad tradeoff.

example: A partial solution may be better than no solution at all.

The configuration of device resources was a major problem for many customers, resulting in numerous support calls. A team had a proposal that could realistically eliminate 95% of the problems ... but the remaining 5% were truly hard. In many cases the committee might decide that a solution that fails 5% of the time is a solution that doesn't work.

In this case, however, we were already shipping a mechanism that had problems 100% of the time, and a mechanism that solved 95% of these problems would clearly be perceived by customers to be a major improvement.

example: It is only reasonable to require a project to use a standard mechanism if the standard mechanism works (or can reasonably be made to work).

A project needed to perform an authentication operation, and attempted to use a standard authentication framework. They knew that there was not yet any support for their authentication protocol, but they then discovered that the framework was fundamentally incapable of supporting this type of authentication.

In most cases, the committee would have required the project team to implement their authentication within the existing framework. In this case, however, it was deemed inappropriate to require the project team to rearchitect the standard framework. Moreover, this case might call into question the wisdom of continuing to require new cases to use the standard framework, and an advisory to the steering committee explained the importance of getting the framework fixed.

example: All products are not intended for all customers, and all customers do not have the same expectations:

A volume manager for production use on servers in large enterprises was intended to be used by tens of thousands of customers who expected it to "just work". This product had to fulfill very high expectations for robustness and ease of use.

A DDI that would be used by third parties to develop and distribute unbundled device drivers required considerable training and skill to use ... so ease of use was far less critical. Interface stability, however, was critical.

A tool to generate code skeletons from MIBs was only likely to be used by a handful of very savvy developers, who could easily work around bugs and accommodate major interface changes from one release to the next.

These projects were (rightly) held to very different standards for robustness, ease of use, and interface stability.

Extending the Scope of a Project

Committees are often confronted with cases that are reasonable as far as they go ... but do not go far enough. When is it reasonable to require a case to do more, and when is it not?

A reasonable case should not be rejected, merely because it is not the case you wish it were. If the case delivers useful functionality in a reasonable architecture, but there are additional problems that could be solved by an extension of the project, this is (in most cases) not adequate grounds for requiring an extension of the project. Rather, the desired follow-on project should be described in the Advisories to the Steering Committee section of the opinion.

There are limits to how much work can be accomplished in one phase of a project. ARCs should not approve incomplete projects (the delivery of useless components), but it is completely reasonable to deliver a long roadmap of features in phased installments. There have been numerous multi-year ``programs'' that were delivered as a series of incremental projects. The key to deciding whether or not a sub-project can be reviewed and approved in isolation is whether or not it is complete; Does this project make sense, and deliver useful functionality, whether or not it is ever followed up by subsequent projects?

If so, the sub-project can be reviewed and approved as a complete project.

If the proposed sub-project is incomplete, it can still be approved, but with a dependency on additional (complementary) projects. This approves the interfaces in the proposed sub-project, but will only permit it to be delivered if it is accompanied by the other projects on which it depends. 

Any complex system is apt to have numerous well known and long standing problems, and new projects often bump into these problems. It is often tempting to require the new project to fix the underlying problem. Occasionally this may be appropriate ... but in some cases it is not:

IF
AND the project does not do anything to make the problem worse, or to cause more users to confront the problem more often
AND the project can work-around the problem so that the customers will not be greatly inconvenienced by it
THEN there may be little basis for "holding the project hostage" over a problem that they neither created nor aggravated.

But, on the other hand, there may be a time to say ``Enough is enough!'':

For years, we introduced a new (and incompatible) frame buffer configuration utility with each new frame-buffer. Each time the ARC approved the case, with an advisory to the steering committee that this was a waste of resources, confusing to our customers, and we should fund a project to create a general frame buffer configuration utility.

After a few years, and a few more configuration utilities, the committee approved another one, with an advisory that this was the last frame buffer specific configuration utility that would be approved.

Six months later, another frame buffer came out and another device specific configuration utility was proposed ... and the team was told that it would not be approved. The project to create a common frame buffer configuration utility was started immediately.