Wiki code for Crossbow Hands-on Steps

Hide Line numbers
1: == // These steps are old and in the process of being updated. 20 Oct 2009 //
2: 
3: ===  Overview of the Crossbow Hands-on Vignette
4: 
5:  Welcome. This session will give you an overview of the key features in Crossbow, Network Virtualization and Resource Control. The version we are using here is currently an add-on to Solaris Neveda build 81, and will be integrated into the main Nevada code base later this yaer, after a beta period that ends late April.
6: 
7:  You need a dedicated Solaris installation as some of the operations, and all system configuration and resource managment, must be done in the global zone. You will configure Crossbow components in the global zone, and then demonstrate them using non-global zones. Zonea are being used only because it is easier to see the effects. An application in the global zone can use a VNIC or have network resource controls in effect.
8: 
9:  The system must have NIC(s) that are GLDv3, such as bge or e1000g. For a full list of NICs, [[see this list>>Project crossbow.#ipinst_which_nic]].
10: 
11: == // Setup //
12: 
13: === Download and Install Crossbow
14: 
15:  Download the SPARC or x86 Crossbow bits from the [[OpenSolaris download site>>Project crossbow.snapshots]]. You have the choice of installing from DVD or using an existing Solaris Nevada build 81 installation and adding the Crossbow bits to that via BFU.
16: 
17:  For the demonstration of the Crossbow features, it is simple to use zones to show the capabilities. Lets start by creating a master zone from which it will be easy to quickly create numerous zones for the different tests.
18: 
19:  Cloning zones is extremely fast when the zonepath is on a ZFS file system. If you have the space available, create a zpool, put a file system on that, and set the mount point to ///zones// just to make it easy. This example has a spare slice available on /dev/dsk/c0t0d0s7. The examples have [[configuration files>>attach:crossbow-vignette.tar]] stored locally in /export/data/zones/.
20: 
21: {{{
22: 
23: **global#** zpool create mypool c0t0d0s7
24: 
25: **global#** zfs create mypool/zones
26: 
27: **global#** zfs set mountpoint=/zones mypool/zones
28: 
29: }}}
30: 
31:  Create a VNIC for the master zone. This is not necessary, as any physical NIC could be used, but why not try out VNICs right away. Lets see what device(s) are available to create a VNIC on top of.
32: 
33: {{{
34: 
35: **global#** dladm show-dev
36: 
37: }}}
38: 
39:  Create a single VNIC. VNIC IDs can range from 0 to 899.
40: 
41: {{{
42: 
43: **global#** dladm create-vnic -d bge0 -m auto 0
44: 
45: }}}
46: 
47:  Configure, install, and boot the zone. Per the [[Crossbow release notes>>http://dlc.sun.com/osol/netvirt/downloads/20080310/release_notes]], on SPARC systems a sparse zone must also inherit /etc/crypto.
48: 
49: {{{
50: 
51: **global#** zonecfg -z zonemaster -f /export/data/zones/zonemaster.cfg
52: 
53: **global#** zoneadm -z zonemaster install
54: 
55: **global#** cp /export/data/zones/zonemaster.sysidcfg /zones/zonemaster/root/etc/sysidcfg
56: 
57: **global#** zoneadm -z zonemaster boot
58: 
59: }}}
60: 
61:  Customize the zone to my preferences.
62: 
63: {{{
64: 
65: **global#** zlogin zonemaster svcadm disable svc:/system/avahi-bridge-dsd:default
66: 
67: **global#** zlogin zonemaster svcadm disable sendmail
68: 
69: **global#** zlogin zonemaster svcadm disable svc:/application/print/ppd-cache-update
70: 
71: **global#** cp /export/data/zones/passwd /zones/zonemaster/root/etc/passwd
72: 
73: **global#** cp /export/data/zones/doc.profile /zones/zonemaster/root/.profile
74: 
75: **global#** cp /export/data/zones/doc.kshrc /zones/zonemaster/root/.kshrc
76: 
77: **global#** zoneadm -z zonemaster halt
78: 
79: **global#** dladm delete-vnic 0
80: 
81: }}}
82: 
83:  The system is ready to use Crossbow.
84: 
85: == // Demonstrating and Using the Crossbow Features //
86: 
87: ===  Tools to Inspect Network Configuration
88: 
89:  The dladm(1M) command displays and sets a variety of network properties. This shows all the network devices configured on the system. dladm has been in Solaris 10 since 1/06, and is also used to manage link properties and aggregations (replaces Sun Trunking). In Solaris Nevada it also manages WiFi, including WEP and WPA security.
90: 
91:  With the integration of Clearview into Nevada in build 83, a number of changes have been made in function and output format. The examples here use features and functions available prior to that.
92: 
93: {{{
94: 
95: **global#** dladm show-dev
96: 
97: }}}
98: 
99:  Show all the data links configured on the system. This will include aggregations, VLANs, VNICs
100: 
101: {{{
102: 
103: **global#** dladm show-link
104: 
105: }}}
106: 
107: ===  Create and Configure Zones with VNICs
108: 
109:  Create a VNIC on device bge0 with an automatically generated MAC address, and give it ID #1. If the NIC has factory MAC addresses available, one of them will be used. Otherwise, a random address is selected. The auto mode is the default action if none is specified.
110: 
111: {{{
112: 
113: **global#** dladm create-vnic -d bge0 -m auto 1
114: 
115: }}}
116: 
117:  Create a second VNIC.
118: 
119: {{{
120: 
121: **global#** dladm create-vnic -d bge0 -m auto 2
122: 
123: }}}
124: 
125:  List all the links now, and see that the VNICs are also shown.
126: 
127: {{{
128: 
129: **global#** dladm show-link
130: 
131: }}}
132: 
133:  List only the virtual NICs.
134: 
135: {{{
136: 
137: **global#** dladm show-vnic
138: 
139: }}}
140: 
141:  Two zones have been created and configured (to save time). Zone //zone1// is assigned the new VNIC //vnic1//, and similarly, zone //zone2// has //vnic2//.
142: 
143: {{{
144: 
145: **global#** zoneadm list -cv
146: 
147: }}}
148: 
149:  Lets configure some zones to use the VNICs just created. To speed this up, a master zone is already set up, so that it can be cloned. The zone path is on ZFS even though Sun does not currently support that configuration. Zone cloning on ZFS is very fast!
150: 
151:  Note that ip-type=exclussive is set and that net only has //physical// set, but not an //address//. For exclusive IP Instance zones, the network configuration is set within the zone, not by the global zone using zonecfg(1M).
152: 
153: {{{
154: 
155: **global#** cat /export/data/zones/zone1.cfg
156: 
157: **global#** zonecfg -z zone1 -f /export/data/zones/zone1.cfg
158: 
159: **global#** zoneadm -z zone1 clone zonemaster
160: 
161: }}}
162: 
163:  Configure, clone, and pre-populate the zone’s /etc/sysidcfg file. One of the administrative features of zones is that you can manage a lot of the non-global zones’ characteristics from the global zone, and this can all be scripted.
164: 
165: {{{
166: 
167: **global#** cat /export/data/zones/zone1.sysidcfg
168: 
169: **global#** cp /export/data/zones/zone1.sysidcfg /zones/zone1/root/etc/sysidcfg
170: 
171: }}}
172: 
173:  Do the same for the second zone.
174: 
175: {{{
176: 
177: **global#** zonecfg -z zone2 -f /export/data/zones/zone2.cfg
178: 
179: **global#** zoneadm -z zone2 clone zonemaster
180: 
181: **global#** cp /export/data/zones/zone2.sysidcfg /zones/zone2/root/etc/sysidcfg
182: 
183: }}}
184: 
185:  Boot both non-global zones.
186: 
187: {{{
188: 
189: **global#** zoneadm -z zone1 boot
190: 
191: **global#** zoneadm -z zone2 boot
192: 
193: }}}
194: 
195:  Put some additional name resolution information into /etc/hosts for each zone
196: 
197: {{{
198: 
199: **global#** cat /export/data/zones/hosts.addon >> /zones/zone1/root/etc/hosts
200: 
201: **global#** cat /export/data/zones/hosts.addon >> /zones/zone2/root/etc/hosts
202: 
203: }}}
204: 
205: ===  Using Zones with VNICs
206: 
207:  Log into the second zone, which is going to be a web server. Use the //-C// option to log in using the console to see any errors and stay connected on a reboot of the zone.
208: 
209: {{{
210: 
211: **global#** zlogin -C zone2
212: 
213: }}}
214: 
215:  FOR THIS LAB: There apprears to be a small issue with some of the systems where the zone’s initial bootup and configuration takes a relatively long time. As a result the console apprears to be hung. So create a second session into the zone while the system does it work.
216: 
217: {{{
218: 
219: **global#** zlogin zone2
220: 
221: }}}
222: 
223:  There is an issue in Nevada build 81 with the SSL configuration of Apache2, so apply a work-around. (It must be **//mv//**, not //cp//.) Then enable the web server via SMF. Next, create a large file and change its ownership to that of the web server processes.
224: 
225: {{{
226: 
227: **zone2#** mv /etc/apache2/2.2/conf.d/ssl.conf /etc/apache2/2.2/conf.d/ssl.conf.saved
228: 
229: **zone2#** svcadm enable apache22
230: 
231: **zone2#** mkfile 100m /var/apache2/2.2/htdocs/largefile
232: 
233: **zone2#** chown webservd:webservd /var/apache2/2.2/htdocs/largefile
234: 
235: }}}
236: 
237:  Test that the web server works locally. Note the speed of the retrieval, especially a second or third time. (NOTE: each time a new file will be created in /tmp, so it can fill up! If running wget a lot, run ’rm /tmp/*file*’ occasionally.)
238: 
239: {{{
240: 
241: **zone2#** cd /tmp
242: 
243: **zone2#** /usr/sfw/bin/wget http://localhost/largefile
244: 
245: }}}
246: 
247:  Log into the first zone.
248: 
249: {{{
250: 
251: **global#** zlogin -C zone1
252: 
253: }}}
254: 
255:  Retrieve the same file via HTTP but from a different zone. Since both zones are using the same physical interface for the VNICs, the traffic stays within the system.
256: 
257: {{{
258: 
259: **zone1#** cd /tmp
260: 
261: **zone1#** /usr/sfw/bin/wget http://zone2/largefile
262: 
263: }}}
264: 
265:  Notice the difference in download speed compared to the previous operation within a single zone. Traffic is now traveling down to the VNIC and back up, not just being looped back in IP.
266: 
267:  Now try the same from the global zone
268: 
269: {{{
270: 
271: **global#** cd /tmp
272: 
273: **global#** /usr/sfw/bin/wget http://zone2/largefile
274: 
275: }}}
276: 
277: ===  Demonstrate VNIC Bandwidth Control
278: 
279:  Reduce the bandwidth for the VNIC associated with the web server zone. Try using 80Mbps since that translates into around 10MBps (Bytes per second), which is easy to see. The dladm command uses bits for its rates. Bandwidth below 10Mbps is not suggested, as the time increments to control the bandwidth are too small. Bandwidth control is done in the kernel, not in hardware.
280: 
281: {{{
282: 
283: **global#** dladm set-linkprop -p maxbw=80mb vnic2
284: 
285: **global#** dladm show-linkprop -c -p maxbw vnic2
286: 
287: }}}
288: 
289: {{{
290: 
291: **zone1#** /usr/sfw/bin/wget http://zone2/largefile
292: 
293: }}}
294: 
295:  Collect some realtime data on the traffic on the VNIC. The //-s// option is for statistics, and //-i// set an interval in seconds. The last argument is the VNIC Id.
296: 
297: {{{
298: 
299: **global#** dladm show-vnic
300: 
301: **global#** dladm show-vnic -s -i 2 2
302: 
303: }}}
304: 
305:  Generate some more traffic.
306: 
307: {{{
308: 
309: **zone1#** /usr/sfw/bin/wget http://zone2/largefile
310: 
311: }}}
312: 
313: ===  Creating a Virtual Network without a Physical Network
314: 
315:  It is possible to create a private network within the system, one that does not have a physical NIC associated with it.
316: 
317:  The first step is to create an //etherstub// as a VNIC anchor.
318: 
319: {{{
320: 
321: **global#** dladm create-etherstub 1
322: 
323: }}}
324: 
325:  Now create more VNICs as was done before on the physical NIC.
326: 
327: {{{
328: 
329: **global#** dladm create-vnic -d etherstub1 11
330: 
331: **global#** dladm create-vnic -d etherstub1 12
332: 
333: }}}
334: 
335:  See how it looks.
336: 
337: {{{
338: 
339: **global#** dladm show-vnic
340: 
341: }}}
342: 
343:  The new NICs need to be assigned to the zones. Using zonecfg interactively.
344: 
345: {{{
346: 
347: **global#** zonecfg -z zone1
348: zonecfg:zone1> add net
349: zonecfg:zone1:net> set physical=vnic11
350: zonecfg:zone1:net> end
351: zonecfg:zone1> exit
352: 
353: }}}
354: 
355:  And a way that can be scripted.
356: 
357: {{{
358: 
359: **global#** zonecfg -z zone2 << EOF
360: add net
361: set physical=vnic12
362: end
363: exit
364: EOF
365: 
366: }}}
367: 
368:  Verify that the confiration changes took affect.
369: 
370: {{{
371: 
372: **global#** zonecfg -z zone1 info net
373: 
374: **global#** zonecfg -z zone2 info net
375: 
376: }}}
377: 
378:  Reboot the zones to add the new devices.
379: 
380: {{{
381: 
382: **global#** zoneadm -z zone1 reboot &
383: 
384: **global#** zoneadm -z zone2 reboot &
385: 
386: }}}
387: 
388:  The dladm(1M) command can not be run in a non-global zone. One way of seeing what network devices are assigned to the zone is to plumb them all. Note that those interface(s) already plumbed will generate an error message.
389: 
390: {{{
391: 
392: **zone1#** ifconfig -a plumb
393: 
394: **zone1#** ifconfig -a
395: 
396: **zone1#** ifconfig vnic11 10.10.10.11/24 up
397: 
398: }}}
399: 
400:  Do the same for the second zone.
401: 
402: {{{
403: 
404: **zone2#** ifconfig -a plumb
405: 
406: **zone2#** ifconfig
407: 
408: **zone2#** ifconfig vnic12 10.10.10.12/24 up
409: 
410: }}}
411: 
412:  Try reaching one zone from the other via the private network and address.
413: 
414: {{{
415: 
416: **zone1#** ping 10.10.10.12
417: 
418: }}}
419: 
420:  It is possible to snoop on any VNIC assigned to a zone, even a private one.
421: 
422: {{{
423: 
424: **zone2#** snoop -d vnic12
425: 
426: }}}
427: 
428: {{{
429: 
430: **zone1#** ping 10.10.10.12
431: 
432: }}}
433: 
434:  Note that this address is not reachable from the global zone, nor from any other system.
435: 
436: {{{
437: 
438: **global#** ping 10.10.10.12
439: 
440: }}}
441: 
442:  Bring the private interface down
443: 
444: {{{
445: 
446: **zone1#** ifconfig vnic1 down unplumb
447: 
448: }}}
449: 
450:  Now that zone can still reached from the other zone.
451: 
452: {{{
453: 
454: **zone2#** ping 10.10.10.11
455: 
456: }}}
457: 
458:  But it can’t be reached from the global zone any more. So the zone is isolated from the outside world, but not from zone //zone2//.
459: 
460: {{{
461: 
462: **global#** ping zone1
463: 
464: }}}
465: 
466: ===  Using Flows
467: 
468:  VNICs have the option of network bandwidth and CPU resource control. Sometimes, finer control based on layers 3 or 4 is desirable. Flows are designed to categorize data by transport (TCP, UDP), IP address, or port number.
469: 
470:  Create a flow definition on bge0 for HTTP traffice (TCP port 80). (NOTE: A link must be plumbed before a flow is created on that link. At this time, flows on VNICs are supported.)
471: 
472: {{{
473: 
474: **global#** flowadm add-flow -l bge0 -a transport=tcp,remote_port=80 tcp80
475: 
476: **global#** flowadm show-flow
477: 
478: }}}
479: 
480:  Turn on accounting.
481: 
482: {{{
483: 
484: **global#** acctadm -e extended -f /var/tmp/net-extended net
485: 
486: }}}
487: 
488:  Generate traffic. Do this several times.
489: 
490: {{{
491: 
492: **global#** /usr/sfw/bin/wget http://webserver/largefile
493: 
494: }}}
495: 
496:  Data will be collected in the file, and can be processed at another time. The flowadm //show-usage// option uses the libexxact(3LIB) extended accounting access library. It is also possible to write custom Perl scripts to process this data.
497: 
498: {{{
499: 
500: **global#** flowadm show-usage -f /var/tmp/net-extended
501: 
502: }}}
503: 
504:  Perform some more wget operations.
505: 
506: {{{
507: 
508: **global#** flowadm show-usage -d -f /var/tmp/net-extended
509: 
510: **global#** flowadm show-usage -f /var/tmp/net-extended tcp80
511: 
512: }}}
513: 
514:  Turn data collection for //net// off.
515: 
516: {{{
517: 
518: **global#** acctadm -x net
519: 
520: }}}
521: 
522: ===  Assigning CPU(s) to a VNIC
523: 
524:  When creating a VNIC or after is is already created, one or more CPUs can be assigned to a VNIC, so that all system processing for the VNIC is done on that set of CPUs. Any one TCP connection will only use one CPU as there is a one-to-one binding of a soft-ring (where data for a connection comes in and goes out of the system) to a CPU. If there are multiple connections, the distribution of CPUs is performed by hashing the TCP port. To demonstrate this, have to global zone terminals open. In one, look at the load distribution across CPUs.
525: 
526: {{{
527: 
528: **global#** mpstat 2
529: 
530: }}}
531: 
532:  Transfer some data across the physical NIC and notice in the mpstat output which CPUs are utilized.
533: 
534: {{{
535: 
536: **zone2#** cd /tmp
537: 
538: **zone2#** /usr/sfw/bin/wget http://webserver/largefile
539: 
540: }}}
541: 
542:  Assign a CPU to do the VNIC processing.
543: 
544: {{{
545: 
546: **global#** dladm set-linkprop -p cpus=23 vnic2
547: 
548: }}}
549: 
550:  Note how CPU 23 now has heavy utilization. Rerun the wget if necessary, or create a loop...
551: 
552: {{{
553: 
554: **zone2#** I=1 ; while [ $I -le 100 ]
555: do
556: rm /tmp/*file*
557: /usr/sfw/bin/wget http://webserver/largefile
558: I=`expr $I + 1`
559: done
560: 
561: }}}
562: 
563:  Assign a different CPU to see the effect better.
564: 
565: {{{
566: 
567: **global#** dladm set-linkprop -p cpus=22 vnic2
568: 
569: }}}
570: 
571:  It is possible to assign a set of CPUs (21,22,23). Multiple streams will be required to show that effect.
572: 
573: {{{
574: 
575: **global#** dladm set-linkprop -p cpus=21,22,23 vnic2
576: 
577: }}}
578: 
579:  The syntax for assigning the CPU when creating the VNIC is as follows.
580: 
581: {{{
582: 
583: **global#** dladm create-vnic -d bge0 -p cpus=21,22,23 -m auto 3
584: 
585: }}}
586: 
587:  Now you are familiar with the basics of Network Virtualization and Resource Control. For more information and copies of DVD images, go to the OpenSolaris Crossbow web site at [[http://www.opensolaris.org/os/project/crossbow/>>Project crossbow.WebHome]].
OpenSolaris

Top Menu

Wiki code for Crossbow Hands-on Steps

Search

Collectives

Community Group

Project

User Group

Subsites

Project crossbow Pages
