Contents

• 1 Automated Install for OpenSolaris 2009.06
• 2 Introduction
  • 2.1 Authors
  • 2.2 Test Sponsor
  • 2.3 Test Plan Approvals
  • 2.4 External Open Source Test Software
  • 2.5 Revision History
  • 2.6 References
  • 2.7 Staffing
  • 2.8 Glossary
• 3 Test Program Summary
• 4 Operational Factors
  • 4.1 Assumptions
  • 4.2 Dependencies
  • 4.3 Risks
• 5 Test Development Information
  • 5.1 Test Development
• 6 Areas of Testing
  • 6.1 Functional Testing
    • 6.1.1 AI web services under SMF
    • 6.1.2 DNS Unicast
    • 6.1.3 Manifests
    • 6.1.4 Retrieve information on running services
  • 6.2 Regression Testing
  • 6.3 Conformance Testing
  • 6.4 Stress/Robustness Testing
    • 6.4.1 Minimal configuration testing
    • 6.4.2 Multiple server/client/subnet testing
    • 6.4.3 Busy network testing
  • 6.5 Performance Testing
  • 6.6 Memory Leak Testing
  • 6.7 Required Solaris Feature Testing
  • 6.8 Interoperability Testing
  • 6.9 Documentation Testing
  • 6.10 Internationalization Testing
  • 6.11 Additional Testing (e.g. BCP, EA Programs)
  • 6.12 Testing Not Covered
• 7 Test Execution
  • 7.1 Hardware Test Configurations
  • 7.2 Software Test Configurations
  • 7.3 Extrapolation Strategy (if applicable)
  • 7.4 Pre-Integration PIT Run
  • 7.5 Test Execution Matrix
• 8 Schedules and Milestones
• 9 Post-Integration Testing Information
  • 9.1 Test Suite Integration Requirements
  • 9.2 Test Suite Execution Integration Requirements
  • 9.3 Other Post-Integration Requirements
• 10 Solaris Update Release (if applicable)

1 Automated Install for OpenSolaris 2009.06

No PSARC ID at this time

2 Introduction

2.1 Authors

2.2 Test Sponsor

2.3 Test Plan Approvals

2.4 External Open Source Test Software

N/A

2.5 Revision History

2.6 References

• Pointer to the test plan: http://www.opensolaris.org/os/project/caiman/auto_install/AI_Testing/AI_tp_2009_06/
• Main OpenSolaris web page for Automated install: http://www.opensolaris.org/os/project/caiman/auto_install/
• Design document: http://www.opensolaris.org/os/project/caiman/auto_install/ai_design/
• Updates to AI design doc for 2009.06: http://www.opensolaris.org/os/project/caiman/auto_install/Design_doc_delta_for_AI_Spring_2009.pdf

2.7 Staffing

2.8 Glossary

3 Test Program Summary

Automated Install was available in a 'preview' version in OpenSolaris 2008.11 for x86 only. The most significant addition to OpenSolaris 2009.spring is SPARC sun4v support. SPARC does not require new tests be written, but does require that existing tests be modified to handle differences (mostly in client booting and where things end up on the boot server) between x86 and SPARC versions.

Other than SPARC support the spring release doesn't add significant new areas of functionality but rather refines and improves areas that were spartan in the preview release. These include changes in how web services are started, DNS support, target options, manifest usability, visibility into services, data formats, and bug fixes. These changes will require moderate enhancements the AI test suites.

No new test suites will be developed. All new tests will be added to the client- or server-side test suites as appropriate.

4 Operational Factors

4.1 Assumptions

This plan covers the test program for the Spring 2009 Automated Install deliverable. As such it only covers functionality expected for the spring release. Testing of functionality for later releases will be documented separately.

4.2 Dependencies

• TBD

4.3 Risks

• Description: Schedule trumps functionality/quality due to hard ship date for OpenSolaris 2009.spring
• Likelihood of Occurrence (High, Medium, Low): High
• Mitigation and Contingency Plan: Work with development team to identify problem areas, determining which must be fixed vs. which can be deferred or have the functionality removed while still delivering a worthwhile product for 2009.spring.

• Description: Availability of test systems with >= 1TB disks to test TB size selection feature of manifests
• Likelihood of Occurrence (High, Medium, Low): Medium
• Mitigation and Contingency Plan: Attempting to order 1.5TB disks to install in test systems. Will attempt to borrow suitable disks for other groups if unable to get our own.

• Description: Churn in supported features up until feature complete deadline
• Likelihood of Occurrence (High, Medium, Low): High
• Mitigation and Contingency Plan: Churn will most likely result in features being dropped rather than added. QE will hold off on detailed test design in areas that seem likely to be dropped or modified.

5 Test Development Information

5.1 Test Development

• Test ability to add/change criteria

6 Areas of Testing

6.1 Functional Testing

6.1.1 AI web services under SMF

• Verify that the Apache web server and AI web server are started as SMF services (started independently of SMF in preview release).
• Verify ability to stop Apache web server and AI web server via SMF.

6.1.2 DNS Unicast

• AI will support DNS unicast in this release but will not set it up. Rather it will inform the user how to do so.
• Verify that AI provides correct instructions for setting up DNS unicast when that option is specified.
• Verify that DNS unicast can be set up via the instructions provided and that the service can be accessed across subnets.

6.1.3 Manifests

• Verify partitions and slices can be selected/controlled via the AI manifest:
  • Flexibility of size units (MB/GB/TB/sectors) for partition and slices.
  • Create/delete/preserve VTOC slices
  • Create/delete fdisk partitions
  • Allow creating non-solaris partitions (Diagnostic, Linux, FAT16, FAT32 and any other simple partitions)
  • Install on to existing slices (remove the restriction of installing to slice 0)
  • Disk selection based on parameters like size, vendor etc.
• Verify software can be customized (package add/delete) via manifests.
• Verification of criteria enhancements
  • Support for NULL criteria in manifests
  • Support adding to or changing criteria of a manifest
  • Verify IP and Mac addresses are now accepted either with colons or without (e.g. 1:e0:c0:12:34:56 vs. 01E0C0123456)
• Verify manifest export (ability to retrieve working manifest for currently published service) functionality
  • Verify exported manifests match the manifests that were used to publish the service
  • Verify exported manifests can be modified and used to republish the same service with differing options or publish a new service.
  • Verify exported manifests contain versioning information
• Verify ability to customize the manifest used as the default

6.1.4 Retrieve information on running services

• Verify supported options for new 'installadm list' sub-command.
• Verify retrieved data matches the request
• Verify retrieved data matches expected values for the services as they were published

6.2 Regression Testing

• Execute existing client- and server-side automated tests that were developed for the preview versions of AI.

6.3 Conformance Testing

• The Automated Install project itself does not have standards that it needs to conform to. Conformance of the resulting Solaris installation to UNIX standards is beyond the scope of this test effort.

6.4 Stress/Robustness Testing

6.4.1 Minimal configuration testing

• Run install tests with the minimal possible configuration -- one system providing all of the server functions (DHCP/TFTP/MDNS/HTTP) for all clients.

6.4.2 Multiple server/client/subnet testing

• Run install tests with each of the server functions (DHCP/TFTP/MDNS/HTTP) running on a dedicated machine, and where allowable have the services/clients spread across multiple subnets.

6.4.3 Busy network testing

• Perform automated installs while the network occupied by the client(s) has heavy traffic from other sources (e.g. NFS, rcp, etc.) to ensure that slower network response times don't cause problems with the install.
• Perform automated installs while the connection to IPS has been disabled to ensure that installation of the base packages occur and appropriate error message(s) are logged about IPS repository.

6.5 Performance Testing

• Compare installation time for a default client/server configuration with that of the same configuration using the 'preview' version of AI from 2008.11. Note that there are no specified performance criteria to be met, so this will be mainly to provide info the the Install development team so they can investigate any unexpected degredation.

6.6 Memory Leak Testing

• This section is primarily concerned with kernel memory leaks. As the only kernel modules used by AI run on the client (which will be rebooted immediately) memory leak testing isn't germane to this project.

6.7 Required Solaris Feature Testing

• Zones Testing

 All AI related server configuration that can be performed within zones (anything dependent on a service like NFS that doesn't run in zones obviously cannot be tested) will be tested on named zones as well as in the global zone. On the client side, AI will not support zones in the 2009.06 release.

 Development is not testing on Zones so Zones testing will be deferred.

• ZFS Testing

 Automated Install uses a ZFS root, so ZFS functionality will be tested indirectly during every install. AI does not make any changes to ZFS, so regression testing of ZFS is not necessary.

• UFS Testing

 The server tools are required to run on Open Solaris via a ZFS file system. UFS testing will be deferred.

• Trusted Extensions Testing

 N/A.

• xVM and LDOM Guest OS Testing

 xVM guest OS testing: The xVM test suite already has the capability to perform automated network installs using jumpstart and do a basic verification that the installed OS reboots and runs properly. Rather than add similar functionality to the AI test suite, the AI test team is partnering with the xVM test team to get AI-installed guest testing added to the xVM test suite. Testing may end up requiring use of both suites -- the AI test suite to set up the servers, and the xVM suite to boot and verify the client.

 LDOM guest testing: Run both server- and client-side tests on LDOM guests. The guest is expected to appear/act like a "bare metal" SPARC system and so all tests should be runnable in an LDOM guest.

6.8 Interoperability Testing

• Test AI client/server configurations with similar architectures on both ends (x86 server with x86 client; sparc server with sparc client) and with differing archetectures (x86 server with sparc client and vice-versa).
• AI needs to work with IPS. The level of interoperability with IPS will be limited to image creation and package installations.

6.9 Documentation Testing

• Test cases and purposes will be designed according to documented manual pages, along with discussions with development. If there are discrepancies, a defect will be filed against development for them to address and dispatch to documentation accordingly. Performed by 'miscellaneous' tests.
• AI network protocol man pages will be used as well and tested indirectly (DHCP, TFTP, DNS, HTTP(S), NFS)

Manual pages:

• installadm

6.10 Internationalization Testing

• No specific effort will be made to test G11N, with exceptions for default settings or due to test team members whose language or locals are outside of the default settings.

6.11 Additional Testing (e.g. BCP, EA Programs)

• There isn't any additional testing requirements for this project.

6.12 Testing Not Covered

• Any AI functionality planned for introduction after the June 2009 release is not covered by this plan and will not be performed as part of this test effort.
• No attempt will be made to achieve exhaustive hardware coverage. The test team will strive to achieve coverage on a variety of processors (Intel, AMD, and sparc4v) and a variety of chipsets (covering various disk, networking, etc. hardware) but will be constrained both by time and the variety of hardware available at the time of the qualification.
• No testing will be done with IPv6 as it is not supported for network booting nor for automated installation in this release.

7 Test Execution

7.1 Hardware Test Configurations

For most tests the AI server and client(s) will be located on the same subnet. However, in order to verify the new functionality that allows clients to access a DNS server on a different subnet, clients and DNS servers will be placed on separate subnets in some cases.

7.2 Software Test Configurations

• IPv4
• The following kernel/library modes: 32/32; 64/32; 64/64
• Mostly N/A, as the AI image controls the version of Solaris and all applications/utilities that will be installed. Testing of software beyond the installation itself and verification of basic OS functionality is beyond the scope of Solaris QE's qualification.

7.3 Extrapolation Strategy (if applicable)

Test configurations will be chosen for the chip sets (ethernet, wifi, disk) they contain. Systems with different vendors/models/processors/etc. will typically not be tested if the graphics/ethernet/wifi chip sets they contain were already tested on a different platform.

7.4 Pre-Integration PIT Run

• Is a pre-integration PIT run required? No. The AI code does not reside in any of the consolidations that are targeted by ON-PIT test suites. If the AI project ends up modifying code in areas that ON-PIT does cover, a project special PIT run will be scheduled to cover the affected areas.

7.5 Test Execution Matrix

The tests summarized in Section 6 will be run on the configurations listed above in Section 7.1. Not all tests will be run on all configurations but all configurations will be covered. Detailed mapping of test suites to configurations and their results will be provided to the ON C-team prior to integration.

8 Schedules and Milestones

9 Post-Integration Testing Information

9.1 Test Suite Integration Requirements

9.2 Test Suite Execution Integration Requirements

9.3 Other Post-Integration Requirements

There are no additional post-integration requirements.

10 Solaris Update Release (if applicable)

This is not applicable due to AI being a replacement for jumpstart and will only be supplied on OpenSolaris. This will not be applied to a Solaris 10 update.