Crossbow Hands-on Steps

 These steps are old and in the process of being updated. 20 Oct 2009 

Overview of the Crossbow Hands-on Vignette

 Welcome. This session will give you an overview of the key features in Crossbow, Network Virtualization and Resource Control. The version we are using here is currently an add-on to Solaris Neveda build 81, and will be integrated into the main Nevada code base later this yaer, after a beta period that ends late April.

 You need a dedicated Solaris installation as some of the operations, and all system configuration and resource managment, must be done in the global zone. You will configure Crossbow components in the global zone, and then demonstrate them using non-global zones. Zonea are being used only because it is easier to see the effects. An application in the global zone can use a VNIC or have network resource controls in effect.

 The system must have NIC(s) that are GLDv3, such as bge or e1000g. For a full list of NICs, see this list.

 Setup 

Download and Install Crossbow

 Download the SPARC or x86 Crossbow bits from the OpenSolaris download site. You have the choice of installing from DVD or using an existing Solaris Nevada build 81 installation and adding the Crossbow bits to that via BFU.

 For the demonstration of the Crossbow features, it is simple to use zones to show the capabilities. Lets start by creating a master zone from which it will be easy to quickly create numerous zones for the different tests.

 Cloning zones is extremely fast when the zonepath is on a ZFS file system. If you have the space available, create a zpool, put a file system on that, and set the mount point to /zones just to make it easy. This example has a spare slice available on /dev/dsk/c0t0d0s7. The examples have configuration files stored locally in /export/data/zones/.

**global#** zpool create mypool c0t0d0s7

**global#** zfs create mypool/zones

**global#** zfs set mountpoint=/zones mypool/zones

 Create a VNIC for the master zone. This is not necessary, as any physical NIC could be used, but why not try out VNICs right away. Lets see what device(s) are available to create a VNIC on top of.

**global#** dladm show-dev

 Create a single VNIC. VNIC IDs can range from 0 to 899.

**global#** dladm create-vnic -d bge0 -m auto 0

 Configure, install, and boot the zone. Per the Crossbow release notes, on SPARC systems a sparse zone must also inherit /etc/crypto.

**global#** zonecfg -z zonemaster -f /export/data/zones/zonemaster.cfg

**global#** zoneadm -z zonemaster install

**global#** cp /export/data/zones/zonemaster.sysidcfg /zones/zonemaster/root/etc/sysidcfg

**global#** zoneadm -z zonemaster boot

 Customize the zone to my preferences.

**global#** zlogin zonemaster svcadm disable svc:/system/avahi-bridge-dsd:default

**global#** zlogin zonemaster svcadm disable sendmail

**global#** zlogin zonemaster svcadm disable svc:/application/print/ppd-cache-update

**global#** cp /export/data/zones/passwd /zones/zonemaster/root/etc/passwd

**global#** cp /export/data/zones/doc.profile /zones/zonemaster/root/.profile

**global#** cp /export/data/zones/doc.kshrc /zones/zonemaster/root/.kshrc

**global#** zoneadm -z zonemaster halt

**global#** dladm delete-vnic 0

 The system is ready to use Crossbow.

 Demonstrating and Using the Crossbow Features 

Tools to Inspect Network Configuration

 The dladm(1M) command displays and sets a variety of network properties. This shows all the network devices configured on the system. dladm has been in Solaris 10 since 1/06, and is also used to manage link properties and aggregations (replaces Sun Trunking). In Solaris Nevada it also manages WiFi, including WEP and WPA security.

 With the integration of Clearview into Nevada in build 83, a number of changes have been made in function and output format. The examples here use features and functions available prior to that.

**global#** dladm show-dev

 Show all the data links configured on the system. This will include aggregations, VLANs, VNICs

**global#** dladm show-link

Create and Configure Zones with VNICs

 Create a VNIC on device bge0 with an automatically generated MAC address, and give it ID #1. If the NIC has factory MAC addresses available, one of them will be used. Otherwise, a random address is selected. The auto mode is the default action if none is specified.

**global#** dladm create-vnic -d bge0 -m auto 1

 Create a second VNIC.

**global#** dladm create-vnic -d bge0 -m auto 2

 List all the links now, and see that the VNICs are also shown.

**global#** dladm show-link

 List only the virtual NICs.

**global#** dladm show-vnic

 Two zones have been created and configured (to save time). Zone zone1 is assigned the new VNIC vnic1, and similarly, zone zone2 has vnic2.

**global#** zoneadm list -cv

 Lets configure some zones to use the VNICs just created. To speed this up, a master zone is already set up, so that it can be cloned. The zone path is on ZFS even though Sun does not currently support that configuration. Zone cloning on ZFS is very fast!

 Note that ip-type=exclussive is set and that net only has physical set, but not an address. For exclusive IP Instance zones, the network configuration is set within the zone, not by the global zone using zonecfg(1M).

**global#** cat /export/data/zones/zone1.cfg

**global#** zonecfg -z zone1 -f /export/data/zones/zone1.cfg

**global#** zoneadm -z zone1 clone zonemaster

 Configure, clone, and pre-populate the zone's /etc/sysidcfg file. One of the administrative features of zones is that you can manage a lot of the non-global zones' characteristics from the global zone, and this can all be scripted.

**global#** cat /export/data/zones/zone1.sysidcfg

**global#** cp /export/data/zones/zone1.sysidcfg /zones/zone1/root/etc/sysidcfg

 Do the same for the second zone.

**global#** zonecfg -z zone2 -f /export/data/zones/zone2.cfg

**global#** zoneadm -z zone2 clone zonemaster

**global#** cp /export/data/zones/zone2.sysidcfg /zones/zone2/root/etc/sysidcfg

 Boot both non-global zones.

**global#** zoneadm -z zone1 boot

**global#** zoneadm -z zone2 boot

 Put some additional name resolution information into /etc/hosts for each zone

**global#** cat /export/data/zones/hosts.addon >> /zones/zone1/root/etc/hosts

**global#** cat /export/data/zones/hosts.addon >> /zones/zone2/root/etc/hosts

Using Zones with VNICs

 Log into the second zone, which is going to be a web server. Use the -C option to log in using the console to see any errors and stay connected on a reboot of the zone.

**global#** zlogin -C zone2

 FOR THIS LAB: There apprears to be a small issue with some of the systems where the zone's initial bootup and configuration takes a relatively long time. As a result the console apprears to be hung. So create a second session into the zone while the system does it work.

**global#** zlogin zone2

 There is an issue in Nevada build 81 with the SSL configuration of Apache2, so apply a work-around. (It must be mv, not cp.) Then enable the web server via SMF. Next, create a large file and change its ownership to that of the web server processes.

**zone2#** mv /etc/apache2/2.2/conf.d/ssl.conf /etc/apache2/2.2/conf.d/ssl.conf.saved

**zone2#** svcadm enable apache22

**zone2#** mkfile 100m /var/apache2/2.2/htdocs/largefile

**zone2#** chown webservd:webservd /var/apache2/2.2/htdocs/largefile

 Test that the web server works locally. Note the speed of the retrieval, especially a second or third time. (NOTE: each time a new file will be created in /tmp, so it can fill up! If running wget a lot, run 'rm /tmp/*file*' occasionally.)

**zone2#** cd /tmp

**zone2#** /usr/sfw/bin/wget http://localhost/largefile

 Log into the first zone.

**global#** zlogin -C zone1

 Retrieve the same file via HTTP but from a different zone. Since both zones are using the same physical interface for the VNICs, the traffic stays within the system.

**zone1#** cd /tmp

**zone1#** /usr/sfw/bin/wget http://zone2/largefile

 Notice the difference in download speed compared to the previous operation within a single zone. Traffic is now traveling down to the VNIC and back up, not just being looped back in IP.

 Now try the same from the global zone

**global#** cd /tmp

**global#** /usr/sfw/bin/wget http://zone2/largefile

Demonstrate VNIC Bandwidth Control

 Reduce the bandwidth for the VNIC associated with the web server zone. Try using 80Mbps since that translates into around 10MBps (Bytes per second), which is easy to see. The dladm command uses bits for its rates. Bandwidth below 10Mbps is not suggested, as the time increments to control the bandwidth are too small. Bandwidth control is done in the kernel, not in hardware.

**global#** dladm set-linkprop -p maxbw=80mb vnic2

**global#** dladm show-linkprop -c -p maxbw vnic2

**zone1#** /usr/sfw/bin/wget http://zone2/largefile

 Collect some realtime data on the traffic on the VNIC. The -s option is for statistics, and -i set an interval in seconds. The last argument is the VNIC Id.

**global#** dladm show-vnic

**global#** dladm show-vnic -s -i 2 2

 Generate some more traffic.

**zone1#** /usr/sfw/bin/wget http://zone2/largefile

Creating a Virtual Network without a Physical Network

 It is possible to create a private network within the system, one that does not have a physical NIC associated with it.

 The first step is to create an etherstub as a VNIC anchor.

**global#** dladm create-etherstub 1

 Now create more VNICs as was done before on the physical NIC.

**global#** dladm create-vnic -d etherstub1 11

**global#** dladm create-vnic -d etherstub1 12

 See how it looks.

**global#** dladm show-vnic

 The new NICs need to be assigned to the zones. Using zonecfg interactively.

**global#** zonecfg -z zone1
zonecfg:zone1> add net
zonecfg:zone1:net> set physical=vnic11
zonecfg:zone1:net> end
zonecfg:zone1> exit

 And a way that can be scripted.

**global#** zonecfg -z zone2 << EOF
add net
set physical=vnic12
end
exit
EOF

 Verify that the confiration changes took affect.

**global#** zonecfg -z zone1 info net

**global#** zonecfg -z zone2 info net

 Reboot the zones to add the new devices.

**global#** zoneadm -z zone1 reboot &

**global#** zoneadm -z zone2 reboot &

 The dladm(1M) command can not be run in a non-global zone. One way of seeing what network devices are assigned to the zone is to plumb them all. Note that those interface(s) already plumbed will generate an error message.

**zone1#** ifconfig -a plumb

**zone1#** ifconfig -a

**zone1#** ifconfig vnic11 10.10.10.11/24 up

 Do the same for the second zone.

**zone2#** ifconfig -a plumb

**zone2#** ifconfig

**zone2#** ifconfig vnic12 10.10.10.12/24 up

 Try reaching one zone from the other via the private network and address.

**zone1#** ping 10.10.10.12

 It is possible to snoop on any VNIC assigned to a zone, even a private one.

**zone2#** snoop -d vnic12

**zone1#** ping 10.10.10.12

 Note that this address is not reachable from the global zone, nor from any other system.

**global#** ping 10.10.10.12

 Bring the private interface down

**zone1#** ifconfig vnic1 down unplumb

 Now that zone can still reached from the other zone.

**zone2#** ping 10.10.10.11

 But it can't be reached from the global zone any more. So the zone is isolated from the outside world, but not from zone zone2.

**global#** ping zone1

Using Flows

 VNICs have the option of network bandwidth and CPU resource control. Sometimes, finer control based on layers 3 or 4 is desirable. Flows are designed to categorize data by transport (TCP, UDP), IP address, or port number.

 Create a flow definition on bge0 for HTTP traffice (TCP port 80). (NOTE: A link must be plumbed before a flow is created on that link. At this time, flows on VNICs are supported.)

**global#** flowadm add-flow -l bge0 -a transport=tcp,remote_port=80 tcp80

**global#** flowadm show-flow

 Turn on accounting.

**global#** acctadm -e extended -f /var/tmp/net-extended net

 Generate traffic. Do this several times.

**global#** /usr/sfw/bin/wget http://webserver/largefile

 Data will be collected in the file, and can be processed at another time. The flowadm show-usage option uses the libexxact(3LIB) extended accounting access library. It is also possible to write custom Perl scripts to process this data.

**global#** flowadm show-usage -f /var/tmp/net-extended

 Perform some more wget operations.

**global#** flowadm show-usage -d -f /var/tmp/net-extended

**global#** flowadm show-usage -f /var/tmp/net-extended tcp80

 Turn data collection for net off.

**global#** acctadm -x net

Assigning CPU(s) to a VNIC

 When creating a VNIC or after is is already created, one or more CPUs can be assigned to a VNIC, so that all system processing for the VNIC is done on that set of CPUs. Any one TCP connection will only use one CPU as there is a one-to-one binding of a soft-ring (where data for a connection comes in and goes out of the system) to a CPU. If there are multiple connections, the distribution of CPUs is performed by hashing the TCP port. To demonstrate this, have to global zone terminals open. In one, look at the load distribution across CPUs.

**global#** mpstat 2

 Transfer some data across the physical NIC and notice in the mpstat output which CPUs are utilized.

**zone2#** cd /tmp

**zone2#** /usr/sfw/bin/wget http://webserver/largefile

 Assign a CPU to do the VNIC processing.

**global#** dladm set-linkprop -p cpus=23 vnic2

 Note how CPU 23 now has heavy utilization. Rerun the wget if necessary, or create a loop...

**zone2#** I=1 ; while [ $I -le 100 ]
do
rm /tmp/*file*
/usr/sfw/bin/wget http://webserver/largefile
I=`expr $I + 1`
done

 Assign a different CPU to see the effect better.

**global#** dladm set-linkprop -p cpus=22 vnic2

 It is possible to assign a set of CPUs (21,22,23). Multiple streams will be required to show that effect.

**global#** dladm set-linkprop -p cpus=21,22,23 vnic2

 The syntax for assigning the CPU when creating the VNIC is as follows.

**global#** dladm create-vnic -d bge0 -p cpus=21,22,23 -m auto 3

 Now you are familiar with the basics of Network Virtualization and Resource Control. For more information and copies of DVD images, go to the OpenSolaris Crossbow web site at http://www.opensolaris.org/os/project/crossbow/.