Open HA Cluster » Documentation » Testing » Testing OHAC
en

Testing OHAC

Testing Open HA Cluster

 This document provides information about how to run Smoke Tests for Open HA Cluster. Smoke Tests include tests for HA Communication Infrastructure and Services Software in Open HA Cluster. For the most up-to-date build instructions, see the Building and Installing the Open HA Cluster Automated Test Suite page. This document contains the following sections:

Configuration Requirements for the Smoke Test Suite

 Before you run the Smoke test suite, ensure that you have met the following prerequisites.

Cluster Node Requirements

  • A cluster with at least two nodes. Both SPARC and x86 configurations are supported.
  • The Open High Availability Cluster (OHAC) software installed on all cluster nodes.
  • The SUNWspro (The Studio Compiler) package installed on all cluster nodes.
  • The Open HA Cluster Automated Test software package (SUNWscts) installed on all cluster nodes. You must provide the path to the C Compiler during installation of the SUNWscts test package.

Top

Client Machine Requirements

  • One external client machine outside the cluster. It can be SPARC or x86.
  • The Open HA Cluster Automated Test software package (SUNWscts) installed on the client machine.

Top

Test Server Machine Requirements

  • One test server machine. It can be SPARC or x86.
  • Open HA Cluster Automated Test software package (SUNWscts) installed on the test server machine.

Top

Individual Test Prerequisites

 For failover and scalable tests:

  • One hostname for configuring a Logical Host or Shared Address resource.
  • One global raw device, either a Solaris Volume Manager metadevice or a /dev/global device for hardware RAID.

 For zone tests:

  • One hostname for configuring a Logical Host or Shared Address resource.
  • One global raw device, either a Solaris Volume Manager metadevice or a /dev/global device for hardware RAID.
  • One non-global zone configured on each cluster node. For information about installation and configuration of a non-global zone, see the System Administration Guide: Virtualization Using the Solaris Operating System page.

NOTE: You can use the same hostname and global raw device in all three types of tests (failover, scalable, and zone). Top

How to Start the SCATE Server Process

 To determine whether the SCATE server process is already running, use one of the following commands:


$ **/usr/scate/svc/bin/scate status**

 or


$ **pgrep scateserver**
  • Log on to the test server as superuser.
  • Start the SCATE server process on the test server, if the SCATE server process is not already running.
# **/usr/scate/svc/bin/scate start**

 Top === How to Create the Test Request File

    1. Log on to the test server as user scate.
    2. Run the smokerun utility from the test server to create a test request file.
$ **/usr/scate/tset/bin/smokerun**

The following is a sample screenshot of a smokerun invocation.

Name of a cluster node ? [phys-foo-1] **pono1**
Obtaining cluster configuration information from node pono1...
Specify Test Results Log Directory that is NFS exported and world writable.
Ex : /home/username/log (or) /net/machine/export/home/log
Test Results Log Directory? **/home/mytest/scate_logs**
Test Request Filename ? [/home/mytest/scate_logs/TestRequest/testrequest1.out] 
~-------------
Test Category
~-------------
        1. Smoke Tests for HA Communication Infrastructure
        2. Smoke Tests for HA Services
Enter the number that corresponds to the tests you want to run.
To select more than one option, Enter a comma seperated list.
Enter your selection : **2**
        1. Failover Services
        2. Scalable Services
        3. Private Interconnect
        4. Non-global Zone
Enter the number that corresponds to the test you want to run.
To select more than one test, you can either enter a comma
seperated list (1,2,3...) or specify a range (1-3) or
enter "all" to run all the tests.
Enter your selection : **1**
***** Generating Entry for Failover Services *****
Some tests you selected require information about Network Hostname. 
Please enter the hostname to be used for configuring Logical Host and Shared Address resource :**ono-1**
Getting node list from the cluster...
Please enter comma seperated node list on which "ono-1" will be hosted [pono1,pono2] : **//Return//**
Validating the Network hostname...
Getting the "NetIfList"(IPMP group info). Please wait...
********************************************************************************
Tests require one global raw device. Device Name should be of the form
/dev/md/foo-1/rdsk/dx for SVM or /dev/global/rdsk/dxsy for a H/W RAID device.
The device should NOT be mounted either globally or locally on any cluster node.
 Also, the device should NOT be mounted at boot thru /etc/vfstab. If the device
is present in /etc/vfstab, set 'mount at boot' field to 'no' on all the nodes.
********************************************************************************
Enter the raw disk slice(s) and end the list with a blank line.
Disk name? **/dev/md/ono-1/rdsk/d0**
Disk name? **//Return//**
You selected 1 disks
Raw Disk 1 = /dev/md/ono-1/rdsk/d0
Is this correct? [y/n] ? **y**
You selected 1 disks
Raw Disk 1 = /dev/md/ono-1/rdsk/d0
Test Request File is in /home/mytest/scate_logs/TestRequest/testrequest1.out
Run the tests using smokerun -f /home/mytest/scate_logs/TestRequest/testrequest1.out

The following is an example test request file for Smoke Failover Services testing:

# SCATE Test Request File
# Generated: n1c33 Thu May 22 07:34:20 PDT 2008
LOGDIR=/home/mytest/scate_logs
CLUSTER
        NODENAME=pono1
END
# Smoke - Failover Services
REQUEST cti
    SEQUENCE=sanity
    SCENARIO=failover
    #SCENARIO=tc_mytest
    #TEST_NUMBER="{1}"
    #TEST_NUMBER="{1-3,5}"
    #TEST_NUMBER=all
    RUN_NEWCLI=TRUE
    NETWORK_HOSTNAME_1=ono-1
    NETWORK_HOSTNAME_ADAPTERS_1="sc_ipmp0@1,sc_ipmp0@2"
    NETWORK_HOSTNAME_HOSTS_1=pono1,pono2
    NETWORK_HOSTNAME_TYPE_1=0
    RAW_DISK_1=/dev/md/ono-1/rdsk/d0
    FS_TYPE=ufs
END

Top === How to Run the Smoke Test
The test request file is created under the log directory that is passed to the smokerun utility. In the previous example, the test request file name is /home/mytest/scate_logs/TestRequest/testrequest1.out. To run the test suite, invoke the following command on the test server as user scate.

$ **/usr/scate/svc/bin/smokerun -f ///path/////testrequestfile//**

 Example:

$ **/usr/scate/svc/bin/smokerun -f /home/mytest/scate_logs/TestRequest/testrequest1.out**
Executing the tests in /home/mytest/scate_logs/TestRequest/testrequest1.out...
Info: (From Server) Host is rsh'able.
Operation Success: (From Server) Job ID is 1
Operation Success: (From Server) Log directory is /home/mytest/scate_logs/05-22-08_09.57.48_pono1_1

Top === How to Run Individual Test Cases==== For CTI Requests

    1. Edit the test request file to add values to the following parameters:
   SCENARIO=tc_//testcase//
   TEST_NUMBER="{//N//}"

 For "{//N//}", specify the numbers of the tests to run by using the following format, including the quotation marks (" ") and the braces ({ }):

"{1}"
Specify a single test
"{1,3}"
Specify multiple tests
"{1-3}"
Specify a range of tests
    1. As user scate, run the smokerun utility, specifying the request file that you edited.
$ **smokerun -f //requestfile//**

For Other Requests

    1. Edit the test request file to add values to the following parameters:
   SCENARIO=tc_//testcase//
   TEST_NUMBER="//N//"

 For "//N//", specify the numbers of the tests to run by using the following format, including the quotation marks (" ") but without the braces you use in CTI request files:

"1"
Specify a single test
"1,3"
Specify multiple tests
"1-3"
Specify a range of tests
    1. As user scate, run the smokerun utility, specifying the request file that you edited.
$ **smokerun -f //requestfile//**

Top === How to Stop the SCATE Server Process

# **/usr/scate/svc/bin/scate stop**

Top === Logs and Reports for the Test Suites The test suites generate the following logs and reports.

    • Test Logs
    • SCATE Report
    • Individual Test Logs==== Test Logs
      The test logs for the smokerun test invocation are created under the log directory that is passed to the smokerun utility. The name of a test log is constructed as follows:
    ///Logdirectory/////timestamp//_//clusternode//_//N///
Logdirectory
The directory that is passed to the smokerun utility
timestamp
Includes the date and time of the test suite invocation
clusternode
The name of the cluster node that is passed to the smokerun utility
N
A running job number
In the previous example, the test logs for the smokerun test invocation are in the /home/mytest/scate_logs/05-22-08_09.57.48_pono1_1/ directory.
Top

SCATE Report A SCATE_REPORT file is created at the top level directory /Logdirectory/timestamp_clusternode_N/SCATE_REPORT/. This file has PASS/FAIL status for all the individual test suites and also a pointer to the detailed logs for the failed test suites.
Top

Individual Test Logs
Test logs for individual entries in the test request file are in the following locations, where //reqno// is a running test request number and //testcase// is the test case name:

Summary Results:
///Logdirectory/////timestamp//_//clusternode//_//N/////reqno//_//testcase///summary
Centralized Log:
CTI: ///Logdirectory/////timestamp//_//clusternode//_//N/////reqno//_//testcase///results/testlog
Other: ///Logdirectory/////timestamp//_//clusternode//_//N/////reqno//_//testcase/////clusternode//.log
Individual Test Logs:
CTI: ///Logdirectory/////timestamp//_//clusternode//_//N/////reqno//_//testcase///results/sorted_results
Other: ///Logdirectory/////timestamp//_//clusternode//_//N/////reqno//_//testcase///*_fi_tests
All the commands that are executed in the Smoke tests, along with the output and the log messages from the code, are captured in the test log files as shown in the following example output.
520|0 1 00123575 1 43|May 22 08:10:35 : Sysid 1, file McExecCmd.c, line 129 : ex
ecute_cmd: gosu ksh -c '/usr/cluster/bin/clrt register SUNW.tsvc2'

 The following are the sample contents of the summary file:

TC /tests/tc_failover
TP 1 tc_failover PASS
TP 2 tc_failover PASS
TP 3 tc_failover PASS

Top === Monitoring Live Logs
You can use the scatestatus -l command to monitor the progress of the smokerun test invocations. The live logs for individual test suites for 'cti' REQUEST entries in the test request file are available in the following location:

    ///Logdirectory/////timestamp//_//clusternode//_//N///1_//xxx///results/0001e/*.AAA

 Use the tail -f command on the live log to show the log messages from the tests that are currently in progress.
If the tests are aborted for some reason, the logs from the tests that were in progress are stored under the live_logs/ subdirectory.
Top

Tags:
Created by admin on 2009/10/26 12:08
Last modified by admin on 2009/10/26 12:08

Collectives

Community Group ha-clusters Pages

XWiki Enterprise 2.7.1.34853 - Documentation