Wiki code for Getting Started

Hide Line numbers
1: ==Getting Started With Crossbow
2: 
3: ===Pre-requisites
4: {{{
5:     - Solaris Nevada build 81 or Solaris Express Developer Release
6:     - Nemo-compliant networking card(bge, e1000g, xge, nxge, ...)
7:     - bfu and acr scripts
8:     - Solaris Networking Virtualization bfu archives
9: }}}
10: 
11: ===Table of Contents
12: -      [[Virtualizing the Networking Devices>>#Virtualizing_the_Networking_Devices]]
13: -      [[Bandwidth Management>>#Bandwidth_Management]]
14: -      [[IP Instances, Exclusive Zones>>#IP_Instances_and_Exclusive_Zones]]
15: -      [[CPU Resources with Network Virtualization>>#CPU_Resources_with_Network_Virtualization]]
16:  
17: == Virtualizing the Networking Devices ==
18: 
19: A single physical networking card is presented as multiple virtual cards,
20: which are called vnics.
21: 
22: A vnic acts like any networking device. It has its own MAC address.
23: An IP interface may be plumbed over a vnic, which can then be
24: assigned an IPv4 or IPv6 address.
25: 
26: 
27: Example:
28: 
29: - List the physical links on the system:
30: 
31: {{{
32:     # dladm show-link
33:     bge0            type: non-vlan  mtu: 1500       device: bge0
34:     ath0            type: non-vlan  mtu: 1500       device: ath0
35: }}}
36: 
37: - Create a virtual NIC over bge0:
38: 
39: {{{
40:     # dladm create-vnic -l bge0 1
41:     # dladm show-vnic
42:     LINK        OVER             SPEED  MACADDRESS         MACADDRTYPE
43:     vnic1       bge0            0 Mbps  2:8:20:22:51:dc    random
44: }}}
45: 
46: Note that a random MAC address was automatically assigned to the VNIC. The VNIC and its MAC address will persist if the host is rebooted.
47: 
48: - A new data link is created:
49: 
50: {{{
51:     # dladm show-link
52:     bge0            type: non-vlan  mtu: 1500       device: bge0
53:     ath0            type: non-vlan  mtu: 1500       device: ath0
54:     vnic1           type: non-vlan  mtu: 1500       device: vnic1
55: }}}
56: 
57: - Bring up an IP interface on vnic1:
58: {{{
59:     # ifconfig vnic1 plumb
60:     # ifconfig vnic1 dhcp start
61:     # ifconfig vnic1
62:     vnic1: flags=201004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4,CoS>
63:         mtu 1500 index 4
64:         inet 129.146.109.26 netmask fffffe00 broadcast 129.146.109.255
65:         ether 2:8:20:22:51:dc
66: }}}
67: 
68: Now, the virtual NIC is visible to all classical network monitoring tools,
69: such as netstat(1M):
70: 
71:         # netstat -ian
72:             Name  Mtu  Net/Dest      Address        Ipkts  Ierrs Opkts  Oerrs Collis Queue 
73:         lo0   8232 127.0.0.0     127.0.0.1      1624   0     1624   0     0      0
74:         lo0   8232 127.0.0.0     127.0.0.1      0      N/A   1618   N/A   N/A    0
75:         ath0  1500 10.192.0.0    10.192.11.51   19036  0     3857   0     0      0
76:         ath0  1500 10.192.0.0    10.192.11.51   17468  N/A   3805   N/A   N/A    0
77:         vnic1 1500 129.146.108.0 129.146.109.26 371    0     12     0     0      0
78:         vnic1 1500 129.146.108.0 129.146.109.26 10     N/A   3      N/A   N/A     0
79: 
80: == Bandwidth Management ==
81: 
82: The bandwidth may be limited for a whole VNIC by specifying the
83: maxbw property (expressed in Mbps) when the vnic is created:
84: 
85: {{{
86:     # dladm create-vnic -l bge0 -p maxbw=15 3
87:     # dladm show-vnic
88:     LINK        OVER             SPEED  MACADDRESS         MACADDRTYPE
89:     vnic1       bge0            0 Mbps  2:8:20:22:51:dc    random
90:     vnic3       bge0            0 Mbps  2:8:20:41:c2:71    random
91:     # dladm show-linkprop -p maxbw vnic3
92:     LINK         PROPERTY        VALUE          DEFAULT        POSSIBLE
93:     vnic3        maxbw           15             ~--             ~--
94: }}}
95: 
96: A bandwidth limit can also be set on an existing data-link such as a physical NIC or an existing VNIC:
97: 
98: {{{
99:     # dladm set-linkprop -p maxbw=300 bge0
100:     # dladm show-linkprop -p maxbw bge0
101:     LINK         PROPERTY        VALUE          DEFAULT        POSSIBLE
102:     bge0         maxbw           300            ~--             ~--
103: }}}
104: 
105: A finer grain limit could be set on a per-transport or on a per-protocol
106: basis, on top of any data-link.
107: For that we use the new command flowadm(1m) to create a new flow of
108: packets defined by the description of the packets matching the flow
109: (transport, local~_port, etc ...). The limit on the bandwidth that can
110: be used by that flow is specified using the maxbw flow property.
111: 
112: {{{
113:     # flowadm add-flow -l vnic2 -a transport=tcp tcp~_flow
114:     # flowadm set-flowprop -p maxbw=100 tcp~_flow
115:     # flowadm show-flow
116:     flow name      flow attributes                         policy attributes        
117:     tcp~_flow       v4:tcp                                  (L)   100 Mbps           
118: }}}
119: 
120: Incoming and outgoing TCP packets will be subject to a maximum of
121: 100Mbps of throughput. Other traffic will use all remaining bandwidth
122: available for vnic1.
123: 
124: We can observe the accumulated statistics about the packets used by
125: the tcp_flow, by running kstat in the global zone:
126: 
127:         bash-3.00# kstat -n tcp~_flow
128:         module: unix                            instance: 0     
129:                 name:   tcp~_flow                        class:    flow
130:                 crtime                          29.903360333
131:                 ierrors                         0
132:                 ipackets                        10177
133:                 obytes                          0
134:                 oerrors                         0
135:                 opackets                        0
136:                 rbytes                          550094
137:                 snaptime                        17969.794047786
138: 
139: Per-flow bandwidth utilization may also be monitored in real time
140: by invoking netstat -K.
141: 
142: In our example with tcp~_flow, this is a snapshot of the real-time
143: output screen, while running [[netperf>>http://www.freebsd.org/projects/netperf/index.html]] between
144: two zones, one attached to vnic1 and the other to vnic2:
145: 
146: 
147:         Flow           Link          iKb/s     oKb/s     iPk/s     oPk/s 
148: 
149:         vnic1          bge0      106236.18    534.43   9068.56   1266.79
150:         vnic2          bge0         534.89 106235.75   1267.78   9067.58
151:         vnic3          bge0           0.46      0.00      0.98      0.00 
152:         vnic4          bge0           0.46      0.00      0.98      0.00
153:         tcp_flow       vnic2        428.95      0.00   1016.77      0.00
154: 
155:         Totals                    13400.12  13346.27  11355.07  10334.38 
156: 
157:  
158: 
159: == IP Instances and Zones ==
160: 
161: A zone with an exclusive IP stack has its own instance of the global tables
162: and variables used in the TCP/IP stack. This allows a zone to be connected to
163: a separate LAN or VLAN without sharing any networking state or policies with
164: other zones.
165: The zone with its exclusive IP instance has its own IP routing table,
166: ARP table, IPsec policies and security associations, IP Filter rules,
167: TCP/IP ndd tunables, etc.
168: 
169: To create a zone with an exclusive IP instance,
170: set ip-type=exclusive in zonecfg(1M).
171: 
172: Since an exclusive zone has control over its ARP and IP internal structures,
173: an IP address does not need to be set for the zone by the global zone
174: administrator any more. 
175: Simply pick a physical interface to be assigned to the zone, 
176: or specify a VNIC which you previously created using dladm(1M).
177: 
178: The following is a sample of zonecfg input:
179: 
180: 
181:         create -b
182:         set zonepath=/export/home/Zones/z3
183:         set autoboot=false
184:         set ip-type=exclusive
185:         add inherit-pkg-dir
186:         set dir=/lib
187:         end
188:         add inherit-pkg-dir
189:         set dir=/platform
190:         end
191:         add inherit-pkg-dir
192:         set dir=/sbin
193:         end
194:         add inherit-pkg-dir
195:         set dir=/usr
196:         end
197:         add inherit-pkg-dir
198:         set dir=/opt
199:         end
200:         add inherit-pkg-dir
201:         set dir=/etc/crypto
202:         end
203:         add net
204:         set physical=vnic3
205:         end
206: 
207: 
208: Copy it into a file (e.g. /var/tmp/zcfg.in).
209: 
210: Then run:
211: 
212: 	# zonecfg -z z3 -f /var/tmp/zcfg.in 
213: 
214: (Note: the following output is normal)
215: >>z3: No such zone configured
216: >>Use ’create’ to begin configuring a new zone.
217: 
218: zoneadm will now show the newly created zone ’z3’ in a configured state:
219: 
220:         # zoneadm list  -cv
221:           ID NAME             STATUS     PATH                           BRAND    IP    
222:            0 global           running    /                              native  shared
223:            1 z1               running    /export/home/Zones/z1          native  excl  
224:            2 z2               running    /export/home/Zones/z2          native  excl  
225:            \- z3               configured /export/home/Zones/z3          native  excl  
226: 
227: Note the ’excl’ under the ’IP’ column indicating an exclusive IP instance for
228: the zone z3.
229: 
230: The zone needs to be installed:
231: 
232:         bash-3.00# zoneadm -z z3 install
233: 
234: Note: depending on the CPU speed, the installation may take around 10 minutes.
235: 
236:         bash-3.00# zoneadm list -cv
237:           ID NAME             STATUS     PATH                           BRAND    IP    
238:            0 global           running    /                              native  shared
239:            1 z1               running    /export/home/Zones/z1          native  excl  
240:            2 z2               running    /export/home/Zones/z2          native  excl  
241:            \- z3               installed  /export/home/Zones/z3          native  excl  
242: 
243: 
244: Boot the zone:
245: 
246:            bash-3.00\# zoneadm -z z3 boot
247: 
248: and connect to the zone’s console by running zlogin -C z3, and initialize the
249: naming services. You will have the usual interactive post installation dialog,
250: set the hostname (zone’s name), the time zone, the zone’s root password, etc ..
251: 
252: 
253: The zone is now ready.
254: You may bring up the network on the zone:
255: 
256:         z3 console login: root
257:         Password: 
258:         Feb 15 10:43:55 z3 login: ROOT LOGIN /dev/console
259:         Sun Microsystems Inc.   SunOS 5.11      snv_55  October 2007
260:         #
261:         # zonename
262:         z3
263:         #
264:         # ifconfig -a plumb
265:         # ifconfig -a
266:         lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
267:                 inet 127.0.0.1 netmask ff000000 
268:         vnic3: flags=201000842<BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2
269:                 inet 0.0.0.0 netmask 0 
270:                 ether 2:8:20:41:c2:71
271: 
272: Note: dladm show-link cannot yet be run from a non global zone. However, ifconfig -a will plumb all non loopback interfaces that were assigned to the zone by the global zone administrator.
273: 
274:         # ifconfig vnic3 1.1.1.3/24 up
275:         # ifconfig -a
276:         lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
277:                 inet 127.0.0.1 netmask ff000000 
278:         vnic3: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2
279:                 inet 1.1.1.3 netmask ffffff00 broadcast 1.1.1.255
280:                 ether 2:8:20:41:c2:71
281: 
282:         # netstat -rn
283: 
284:         Routing Table: IPv4
285:           Destination           Gateway           Flags  Ref     Use     Interface 
286:         ~----~----~----~----~----~----
287:         1.1.1.0              1.1.1.3              U         1          1 vnic3     
288:         224.0.0.0            127.0.0.1            U         1          0 lo0       
289:         127.0.0.1            127.0.0.1            UH        1         36 lo0       
290: 
291: 
292: The zone can now communicate with other hosts on the network.
293: 
294:         # ping 1.1.1.1
295:         1.1.1.1 is alive
296: 
297: Note:
298: Just like a single zoned system, /etc/hostname.vnic3 or /etc/dhcp.vnic3 can
299: be used to have the network automatically initialized with a static or dynamic
300: IP address upon reboot of the zone. 
301: 
302: Back to the global zone:
303: 
304:         bash-3.00# ifconfig -a
305:         lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
306:                 inet 127.0.0.1 netmask ff000000 
307:         ath0: flags=201004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4,CoS> mtu 1500 index 3
308:                 inet 192.168.1.132 netmask ffffff00 broadcast 192.168.1.255
309:                 ether 0:b:6b:4d:b1:4 
310:         bash-3.00# 
311:         bash-3.00# netstat -rn
312: 
313:         Routing Table: IPv4
314:           Destination           Gateway           Flags  Ref     Use     Interface 
315:         ~----~----~----~----~----~----
316:         default              192.168.1.1          UG        1         45 ath0      
317:         192.168.1.0          192.168.1.132        U         1          4 ath0      
318:         224.0.0.0            192.168.1.132        U         1          0 ath0      
319:         127.0.0.1            127.0.0.1            UH        2         68 lo0
320: 
321: The IP interface ’vnic3’ is not exposed to the global zone, or other
322: zones.
323: 
324: Further details about IP instances may be found in the [[Presentation to OpenSolaris User Group>>attach:Project crossbow.Docs@ipinstances-sug1.pdf]]
325: and the [[IP instances Architecture>>attach:Project crossbow.Docs@si-interfaces.pdf]].
326: 
327: 
328: == CPU Resources with Network Virtualization ==
329: 
330: 
331: NICs and VNICs may be bound to a subset of the processors available on a
332: system. When such binding is established, most of the packet processing will be executed on the bound CPUs.
333: 
334: This feature is particularly useful for deeper separation of the CPU resources
335: between zones and containers. It extends that separation to account for most
336: of the CPU cycles spent anonymously in the networking subsystem on behalf of
337: a zone.
338: 
339: The example below illustrates a setup that shows the difference
340: in CPU utilization when this feature is in use.
341: 
342: The NIC will still interrupt a system defined CPU(s), which are currently
343: not under the virtualization control. However, incoming packets are
344: quickly dispatched to be processed by a CPU from the zone’s processor set.
345: 
346: The instructions for creating a CPU resource pool with 4 CPUs out of a T-10000 multi-core system are described [[here>>attach:Project crossbow.FeaturesOverview@PoolSetup.txt]]
347: 
348: Configure a zone work1zone with an exclusive IP stack, and connect it to a
349: vnic called vnic1, as described above. (In this example, we
350: use vnic1 over bge1, which was assigned by the system to interrupt the CPU 8).
351: 
352: Assign the work1-pool pool to work1zone:
353: 
354:         # zonecfg -z work1zone set pool=work1-pool
355: 
356: Reboot work1zone so that it binds to the new pool:
357: 
358: 
359:         # zlogin work1zone init 6
360: 
361: 
362: Running [[iperf>>http://sourceforge.net/projects/iperf]] between the work1zone
363: container and an external host, will actually show that
364: processors 4-7 are mainly used. Most CPUs outside that set are occasionally
365: used at a 5% or less of the time, for other internal load.
366: This is the expected behavior because the application (and all its system
367: calls) are actually hosted by the container.
368: 
369:         CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
370:           0    0   0  106   277  175    0    0    0    2    0     0    0   1   0  99
371:           1    0   0    0     1    0    0    0    0    0    0     0    0   0   0 100
372:           2    0   0    0     1    0    0    0    0    0    0     0    0   0   0 100
373:           3    0   0    0     1    0    0    0    0    0    0     0    0   0   0 100
374:           4    0   0    0  3765    0 7496    7   76  355    0  4556    2  19   0  79
375:           5    0   0    0  3590    0 7175    7   77  336    0  4310    2  19   0  80
376:           6    0   0    0  1504    0 3009    3   15  137    0  1912    1   8   0  92
377:           7    0   0    0  1108    8 2196    1    6   91    0  1394    1   6   0  94
378:           8    0   0 7247  7711 7706    9    0    0  860    0     0    0  56   0  44
379:         ...
380:          22    0   0 9114  6688    0 13659    0    0 1176    0     0    0  64   0  36
381:          23    0   0    0     3    0    5    0    0    0    0     1    0   0   0 100
382: 
383: CPU # 22 was being used at 64% of the time, all inside the
384: system. That is the cost of processing incoming packets for work1zone.
385: That cost is being charged to a CPU that is //not// member of the processor set 
386: that the container is assigned to.
387: 
388: To minimize the unfair utilization of CPU resources induced by work1zone
389: inbound traffic, vnic1 needs to be also "bound" to the CPUs of
390: work1-pset:
391: 
392: From the global zone, run:
393: 
394:         # dladm set-linkprop -p cpus=4,5,6,7 vnic1
395: 
396: 
397: A second experiment with iperf shows:
398: 
399: 
400:         CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
401:           0    0   0  118   270  168    0    0    0    1    0     0    0   0   0 100
402:           1    0   0    0     1    0    0    0    0    0    0     0    0   0   0 100
403:           2    0   0    0     1    0    0    0    0    0    0     0    0   0   0 100
404:           3    0   0    0     1    0    0    0    0    0    0     0    0   0   0 100
405:           4    0   0 2370  4477    0 9045   30   95  623    0  2799    1  33   0  66
406:           5    0   0 2702  4630    0 9366   26   88  608    0  2331    1  32   0  67
407:           6    0   0 2995  4459    0 9017   27   83  737    0  1778    1  34   0  65
408:           7    0   0    0  3225    5 6491    3   51  167    0  4494    2  20   0  79
409:           8    0   0 7574  7587 7574   26    0    1 2767    0     5    0  51   0  49
410:         ...
411:          22    0   0 1243  1088    0 2175    0    0  344    0     4    0   2   0  98
412:          23    0   0    2     8    0   15    0    0    1    0     0    0   0   0 100
413: 
414: Note in particular:
415: . CPUs 4-7 idle time decreased, and their sys time has increased since
416:   they work more to handle work1zone’s packets.
417: . CPU #22 is barely working at 2% instead of the previous 64%.
418:
OpenSolaris

Top Menu

Wiki code for Getting Started

Search

Collectives

Community Group

Project

User Group

Subsites

Project crossbow Pages
