Wiki code for FAQ

Hide Line numbers
1: =  Network Virtualization and Resource Control (Crossbow) FAQ
2: 
3: ===  Table of Contents
4: 
5: ==== [[ Overview >>#overview]]
6: 
7: * [[ What is Crossbow? >>#oview_what]]
8: * [[ When will Crossbow be available? >>#oview_when]]
9: * [[ How do I get Crossbow? >>#oview_how]]
10: 
11: ==== [[ IP Instances >>#ip_instances]]
12: 
13: * [[ What are IP Instances? >>#ipinst_what]]
14: * [[ How do I get them? >>#ipinst_how]]
15: * [[ When will they be available in Solaris 10? >>#ipinst_when]]
16: * [[ Can any NIC be used with IP Instances? >>#ipinst_any_nic]]
17: * [[ Which NICs are known to work with IP Instances? >>#ipinst_which_nic]]
18: * [[ Which NICs don’t work with IP Instances? >>#ipinst_which_nic_not]]
19: * [[ Can I have more than one IP Instance on a system? >>#ipinst_morethanone]]
20: * [[ How many IP Instances are supported? >>#ipinst_howmany]]
21: * [[ How many IP Instances can a non-global zone have? >>#ipinst_howmany_zone]]
22: * [[ Can an IP Instance be shared between two non-global zones? >>#ipinst_shared]]
23: * [[ Can the global zone see into another zone’s IP Instance? >>#ipinst_seeinto]]
24: * [[ How can any of the interfaces assigned to a non-global zone be seen? >>#ipinst_howseen]]
25: * [[ I have configured a zone with ip-type=exclusive, however I don’t see the interface in the zone when running ifconfig. >>#ipinst_ifconfig]]
26: * [[ How do I configure an IP address for a zone? >>#ipinst_hostname]]
27: * [[ What services can now run in a zone that could not before? >>#ipinst_services]]
28: * [[ How do I configure the non-global zone to be a DHCP client? >>#ipinst_dhcpclient]]
29: * [[ What about NFS in a non-global zone? >>#ipinst_nfs]]
30: * [[ Is there anything else that does not work? >>#ipinst_whatwontwork]]
31: 
32: ==== [[ Virtual NICs (VNICs) >>#virtual_nics]]
33: 
34: * [[ What is a VNIC? >>#vnic_what]]
35: * [[ Which NICs support VNICs? >>#vnic_which]]
36: * [[ How many VNICs can I create per NIC? >>#vnic_howmany]]
37: * [[ How many VNICs can I create on a system? >>#vnic_total]]
38: * [[ How is a VNIC managed? >>#vnic_managed]]
39: * [[ How do I dermine whether the NIC has its own set MAC addresses? >>#vnic_ownmac]]
40: * [[ How do I use a factory provided MAC address? >>#vnic_factorymac]]
41: * [[ How do I set my own MAC address? >>#vnic_setownmac]]
42: * [[ Are there restrictions regarding the MAC addresses I can use? >>#vnic_macrestrictions]]
43: * [[ Is it necessary to always specify the MAC address of a VNIC? >>#vnic_whymac]]
44: * [[ Can I use IPv6 addresses with a VNIC? >>#vnic_ipv6]]
45: * [[ Can different VNICs have different MTUs set? >>#vnic_mtu]]
46: * [[ When will VNICs be available in Solaris 10? >>#vnic_s10]]
47: 
48: ==== [[ Flow Management >>#flow_management]]
49: 
50: * [[ What is flow management? >>#flow_what]]
51: * [[ How do I define a flow? >>#flow_how]]
52: * [[ What parameters control a flow? >>#flow_whatparams]]
53: * [[ Which attributes can I use to define the flow? >>#flow_attributes]]
54: * [[ What parameters can be set on flows? >>#flow_parameters]]
55: * [[ Do I have to set bandwidth limits or guarentees? >>#flow_limits]]
56: 
57: ==== [[ Miscellaneous >>#misc]]
58: 
59: * [[ Which NICs have hardware clasification capabilities? >>#misc_nics_with_hw_class]]
60: * [[ How can I tell whether my NIC has hardware receive ring classification capabilities? >>#misc_hw_rings]]
61: 
62: ===  Overview
63: 
64: * What is Crossbow?
65: 
66: Crossbow is a set of technologies that provide network virtualization and greatly improves the resource control, performance, and network utilization needed to achieve true OS virtualization, utility computing, and server consolidation.
67: 
68: Crossbow consists of multiple components:
69: 
70: >* [[ IP Instances >>#ip_instances]]
71: * [[ Virtual NICs (VNICs) >>#virtual_nics]]
72: * [[ Flow Management >>#flow_management]]
73: 
74:  Crossbow is designed to add network virtualization to Solaris without introducing any performance penalty. Some of the underlying work is delivering better network performance. Receive rings and hardware classification, and multiple MAC address support contribute to better performance and enhance the virtualization provided.
75: 
76:  Flow management could introduce some overhead. If the NIC or VNIC is not doing any fanout and B/W control, Crossbow can map a flow directly to a receive (RX) ring and use hardware to classify it. In that case, there is no performance impact. In cases where the NIC or VNIC is already doing bandwidth control or traffic fanout across multiple CPUs, any flow configured on top will have to go through an additional classification layer and there will be a small performance hit.
77: 
78: * When will Crossbow be available?
79: 
80: Crossbow is available in OpenSolaris 2009.06, Solaris Express Community Edition (SX-CE) build 105 and later, or prior versions of OpenSolaris that have been updated with the most recent release or development packages.
81: 
82: * How do I get Crossbow?
83: 
84: Crossbow is installed with OpenSolaris 2009.06 or an update of the packages of an earlier release of OpenSolaris. Installing SX-CE build 105 or later also includes the features.
85: 
86: ===  IP Instances
87: 
88: * What are IP Instances?
89: 
90: IP Instances are separate views of the IP stack, so that visibility and control is limited to the entity (zone) that the instance is assigned to. By default, all of Solaris has one view of IP, and therefor central visibility and control. With zones, the ability to view and control is limited by privileges, and all zones’ network traffic decisions are made with a global view by the kernel. When IP instances are used, the view is limited to that information that applies to the instance, not the full kernel. So routing decisions, for example, are made based on the information only in this instance, and does not use any of the additional information that other instances on the same kernel may have. Similarly, control is delegated to this instance, so that a non-global zone can set network parameters such as routes, ndd(1m) values, IP address(es). Snooping of the interface(s) in the IP Instance is also possible. There is no visibility into any of the other IP Instances that may be sharing this Solaris instance and kernel.
91: 
92:  Another feature with IP Instances is that traffic between zones must pass the whole path down the stack to the underlying NIC or VNIC. This is the result of the zone’s IP not knowing where the destination address is, and outgoing data must thus be put on the wire. If the zone is using a VNIC, whether the traffic stays within the system or exists on a physical netowrk interface depends on whether the destination also using a VNIC sharing the same physical NIC. If a NIC is shared for VNICs, traffic directly between the VNICs will be switched by the VNICs’ virtual switch to the destination VNIC, and it will not leave the system.
93: 
94: * How do I get them?
95: 
96:  IP Instances are in Solaris Nevada build 57 and later.
97: 
98: * When will they be available in Solaris 10?
99: 
100: IP Instances are in Solaris 10 8/07 released on 4 September 2007.
101: 
102: * Can any NIC be used with IP Instances?
103: 
104: All NICs in OpenSolaris and in Solaris Express Community Edition build 80 or later work with IP Instances. For Solaris 10, only NICs supported by the Generic LAN Driver version 3 (GLDv3) are supported with IP Instances. The way to determine if a NIC is GLDv3, run the dladm(1m) command with the ’show-link’ subcommand and look for links that are not of type ’legacy’.
105: 
106:  There is one exception. The **ce** interfaces can also be used now. See [[Which NICs are known to work with IP Instances?>>Project crossbow.#ipinst_which_nic]] for details, such as Nevada build and Solaris 10 patches required.
107: 
108: This is how non-GLDv3 interfaces will look. (Solaris 10 only)
109: 
110: {{{
111: 
112: # dladm show-link
113: eri0            type: legacy    mtu: 1500       device: eri0
114: qfe0            type: legacy    mtu: 1500       device: qfe0
115: qfe1            type: legacy    mtu: 1500       device: qfe1
116: qfe2            type: legacy    mtu: 1500       device: qfe2
117: qfe3            type: legacy    mtu: 1500       device: qfe3
118: 
119: }}}
120: 
121: And how GLDv3 interfaces look.
122: 
123: {{{
124: 
125: # dladm show-link
126: bge0            type: non-vlan  mtu: 1500       device: bge0
127: bge1            type: non-vlan  mtu: 1500       device: bge1
128: bge1001         type: vlan 1    mtu: 1500       device: bge1
129: bge2001         type: vlan 2    mtu: 1500       device: bge1
130: bge2            type: non-vlan  mtu: 1500       device: bge2
131: bge3            type: non-vlan  mtu: 1500       device: bge3
132: aggr1           type: non-vlan  mtu: 1500       aggregation: key 1
133: 
134: }}}
135: 
136: * Which NICs are known to work with IP Instances?
137: ** afe (Nevada build 73 and later)
138: ** bge
139: ** ce (Nevada build 80 and later, Solaris 10 with IP Instance patches*)
140: ** dmfe (Nevada build 73 and later)
141: ** e1000g
142: ** eri (Nevada build 73 and later)
143: ** hme (Nevada build 73 and later)
144: ** iprb (Nevada build 73 and later)
145: ** ixgb
146: ** mxfe (Nevada build 73 and later)
147: ** nge
148: ** nxge
149: ** qfe (Nevada build 73 and later)
150: ** rge
151: ** rtls (Nevada build 73 and later)
152: ** xge
153: ** ath (Nevada only)
154:  * NOTE: The **ce** NIC is not a GLDv3 device, but has been made to work with IP Instances. The patches required for Solaris 10 8/07 are:
155: ** 118777-12 and 137042-01 (SPARC)
156: ** 118778-11 and 137043-01 (i386, x86, x64)
157: 
158: * Which NICs don’t work with IP Instances?
159: ** ce (Nevada build 79 and earlier, and Solaris 10 8/07 or earlier without the patches listed above)
160: ** dnet
161: ** elx
162: ** fjgi
163: ** ge
164: ** ipge
165: ** ixge
166: ** spwr
167:  * NOTE: The e1000g driver replaces ipge in Solaris 10 11/06 and later for these NICs:
168: ** Sun PCI-Express Dual Gigabit Ethernet UTP X7280A-2
169: ** Sun PCI-Express Dual Gigabit Ethernet MMF X7281A-2
170: 
171: The Nemo Unification within Project Clearview has delivered a shim that allows non-GLDv3 interfaces to be used with all the Crossbow and IP Instance features. All releases of OpenSolaris and SX-CE since build 83 have this.
172: 
173: (Solaris 10 only) There are two Change Requests to enable IP Instances with the **ce** driver. See [[What’s Up ce-Doc?>>http://blogs.sun.com/stw/entry/what_s_up_ce_doc]] for some details. These fixes have been put into OpenSolaris and are available in Nevada build 80 and later, and available for Solaris 10 with the patches listed above.
174: 
175: * Can I have more than one IP Instance on a system?
176: 
177: Yes.
178: 
179: * How many IP Instances are supported?
180: 
181: There is no set limit on the number of IP Instances. Since each IP Instance needs (for connectivity) at least one data link (NIC, VLAN-tagged NIC, aggregation, or VNIC), the limit is essentially determined by the number of data links available and manageable on the system.
182: 
183: * How many IP Instances can a non-global zone have?
184: 
185: A non-global zone can have only one IP Instance. By default, a zone is in the global instance sharing IP with the global zone and all other zones without an exclusive IP Instance. When a zone is configured to have an exclusive IP Instance, its view of IP is now isolated from the rest of the system.
186: 
187: * Can an IP Instance be shared between two non-global zones?
188: 
189: No.
190: 
191: * Can the global zone see into another zone’s IP Instance?
192: 
193: Commands at the IP level such as {{code}}ifconfig(1m){{/code}} will only work with the interfaces in the IP Instance from which the command is run. In the global zone, they will only be able to see those interfaces **not** set as exclusive to a non-global zone.
194: 
195: The {{code}}snoop(1m){{/code}} command is able to be used from the global zone even if that interface has been given to a non-global zone with an IP Instance configured. If snoop is run in the global zone and in the zone that has exclusive access to the interface, they will see the same data.
196: 
197: The {{code}}dladm(1m){{/code}} command is used from the global zone to manage all devices, links, aggregations, VLANs, and VNICs. At this time, dladm can not be used in a non-global zone, even if it is configured as an exclusive zone.
198: 
199: * How can any of the interfaces assigned to a non-global zone be seen?
200: 
201: Using the {{code}}dladm(1m){{/code}} command, a privileged user in the global zone can see and control the physical interfaces (NICs, link aggregations (aggr), VLANs, and VNICs).
202: 
203: * I have configured a zone with ip-type=exclusive, however I don’t see the interface in the zone when running ifconfig.
204: 
205: All interfaces assigned to a non-global zone can be identified by running ’ifconfig -a plumb’, followed by ’ifconfig -a’.
206: 
207: {{{
208: non-global zone# ifconfig -a
209: lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
210:     inet 127.0.0.1 netmask ff000000
211: non-global zone# ifconfig -a plumb
212: non-global zone# ifconfig -a
213: lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
214:     inet 127.0.0.1 netmask ff000000
215: bge1: flags=201000842<BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2
216:     inet 0.0.0.0 netmask 0
217:     ether 0:3:ba:e3:42:8c
218: non-global zone#
219: }}}
220: 
221: * How do I configure an IP address for a zone?
222: 
223:  If you have, for example, an nge interface, one method is to create the file /etc/hostname.nge0 in the non-global zone.
224: 
225: {{{
226: non-global zone# echo "192.168.1.11/24" > /etc/hostname.nge0
227: non-globalzone# init 6
228: ...
229: non-global zone# ifconfig -a
230: lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
231: nge0: flags=1000843"<"UP,BROADCAST,RUNNING,MULTICAST,IPv4">" mtu 1500 index 2
232:         inet 192.168.1.11 netmask ffffff00 broadcast 10.1.14.255
233:         ether 0:17:31:46:d8:eb
234: non-global zone#
235: }}}
236: 
237:  Generally, you will set up the /etc/hosts file, /etc/defaultrouter if using default routes, /etc/netmasks, /etc/resolv.conf, and the like, as with any stand-alone system. With a shared IP Instance, much of this is managed by the adminstrator(s) in the global zone.
238: 
239: * What services can now run in a zone that could not before?
240: ** DHCP client
241: ** DHCP server
242: ** Routing daemon
243: ** IPsec
244: ** IPfilter
245: ** IP Multipathing (IPMP)
246: ** ndd commands
247: ** ifconfig (with set or modify capabilities)
248: 
249: * How do I configure the non-global zone to be a DHCP client?
250: 
251:  After configuring and installing the zone, copy or create an /etc/sysidcfg file. For example,
252: 
253: {{{
254: global-zone# cat /myzones/dhcpzone/root/etc/sysidcfg
255: system_locale=C
256: terminal=xterm
257: network_interface=primary {
258:         dhcp
259:         protocol_ipv6=no
260: }
261: security_policy=NONE
262: name_service=NONE
263: nfs4_domain=dynamic
264: timezone=US/Eastern
265: root_password=""
266: global-zone# zlogin dhcpzone ifconfig -a
267: lo0: flags=2001000849"<"UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL">" mtu 8232 index 1
268:         inet 127.0.0.1 netmask ff000000
269: bge2: flags=201004843"<"UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4,CoS">" mtu 1500 index 2
270:         inet 10.1.14.161 netmask ffffffc0 broadcast 10.1.14.191
271:         ether 0:3:ba:e3:42:8d
272: global-zone#
273: 
274: }}}
275: 
276:  Or, you can manually bring up the interface with DHCP.
277: 
278: {{{
279: 
280: dhcp-zone# ifconfig bge2 plumb
281: dhcp-zone# ifconfig bge2 auto-dhcp
282: 
283: }}}
284: 
285:  To have it use DHCP on subsequent boot ups.
286: 
287: {{{
288: 
289: dhcp-zone# touch /etc/hostname.bge2
290: dhcp-zone# touch /etc/dhcp.bge2
291: 
292: }}}
293: 
294: * What about NFS in a non-global zone?
295: 
296: A non-global zone can still be an NFS client (not of the global zone on the same system), but can not be an NFS server. The in-ability of a non-global zone to be an NFS server is not related to networking, but rather to file system and virtual memory interaction. Whether or not IP Instances, VNICs, or any other Crossbow feature is used does not affect or change this situation.
297: 
298: * Is there anything else that does not work?
299: 
300: You can not load private kernel modules in a non-global zone, even if you have your own instance. Also, IPfilter rulesets are controlled from the global zone at this time. A linux branded zone does not work with IP Instances at this time.
301: 
302: ===  Virtual NICs (VNICs)
303: 
304: * What is a VNIC?
305: 
306: A VNIC is a virtualized network interface that presents the same media access control (MAC) interface that an actual interface would provide. Multiple VNICs can be configured on top of the same interface, allowing multiple consumers to share that interfaces. If the interface has hardware clasification capabilities, when data arrives on the NIC, the hardware can automatically direct the datagrams to receiver buffers (rings) associated with a specific VNIC. It may be possible to selectively turn interrupts on and off per ring, allowing the host to control the rate of arrival of packets into the system. For hardware that does not have these capabilities, these features are provided via software.
307: 
308: * Which NICs support VNICs?
309: 
310: VNICs are support on Generic LAN Driver version 3 (GLDv3) supported interfaces. For a list, see [[ Which NICs are known to work with IP Instances?>>#ipinst_which_nic]] You can also create a VNIC on top of an aggregation or VLAN that is built using GLDv3 NICs.
311: 
312: * How many VNICs can I create per NIC?
313: 
314: There is no limit to the number VNICs per NIC. However, for NICs with hardware classification capabilities, maximum performance is achieved when the number of VNICs does not exceed the number of hardware classifiers on the NIC.
315: 
316: * How many VNICs can I create on a system?
317: 
318: There is no maximum number of VNICs on a system (there use to be and it has been removed). As is typically the case, each VNIC will require additional system resources such as CPU. So there will be a practical maximum per system based on the type of system, the type of NIC(s), and traffic patterns.
319: 
320: * How is a VNIC managed?
321: 
322: The dladm(1m) command is used to create, modify, and delete VNICs.
323: 
324: * How do I dermine whether the NIC has its own set of factory-provided MAC addresses?
325: 
326: Use {{code}} dladm show-dev -m  {{/code}} to show the MAC addresses assigned to this NIC.
327: 
328: * How do I use a factory provided MAC address?
329: 
330: To use a factory provided MAC address, run {{code}} dladm create-vnic -m factory  {{/code}} when creating a VNIC.
331: 
332: * How do I set my own MAC address?
333: 
334: The MAC address for a VNIC can be set when the VNIC is created with the {{code}} dladm create-vnic -m MAC-address {{/code}} command in the global zone.
335: 
336:  Future???: The MAC address can be modified using the ifconfig(1m) command in a non-global zone. Either operation must be done by a privileged administrator.
337: 
338: * Are there restrictions regarding the MAC addresses I can use?
339: 
340: Yes, the MAC address must be a valid MAC address as per IEEE. It can not be a multicast or broadcast address.
341: 
342: * Is it necessary to always specify the MAC address of a VNIC?
343: 
344: No, you can choose to use a factory MAC address, have the system generate a random MAC address, or provide one. The default is to use an available factory address, and if none is available, generate a random one.
345: 
346: * Can I use IPv6 addresses with a VNIC?
347: 
348: Yes. You can do most of the things that you can do with a physical NIC. Things you can not do with a VNIC include: create a link aggregation, set a frame size larger than the underlink link.
349: 
350: * Can different VNICs have different MTUs set?
351: 
352: Yes, but no larger than the MTU allowed by the underlying NIC.
353: 
354: * When will VNICs be available in Solaris 10?
355: 
356: At this time there is no specific plan to back port VNICs to Solaris 10. If you have a need for VNIC support in Solaris 10, please add a service record to change request (CR) 6790102.
357: 
358: === Flow Management
359: 
360: * What is flow management?
361: 
362: Flow management is the ability to manage networking bandwidth resource for a transport, service, or a virtual machine. A service is specified as a combination of transport (e.g. TCP, UDP) and port, while a virtual machine is specified by its mac address or an IP address.
363: 
364: * How do I define a flow?
365: 
366: Flows are managed with the {{code}} flowadm(1m) {{/code}} command.
367: 
368: * What parameters control a flow?
369: 
370: Flows are defined as a set of attributes based on Layer 2, 3, and 4 headers which can be used to identify a protocol, service, or a [virtual machine] instance, such as a zone or Xen domain.
371: 
372: * Which attributes can I use to define the flow?
373: ** local_mac_addr
374: ** SAP
375: ** local_ip_addr
376: ** transport (TCP, UDP, or SCTP)
377: ** local_port
378: ** remote_port
379: 
380: * What parameters can be set on flows?
381: 
382: Flows support the following parameters:
383: 
384: >* Bandwidth~--Sets the full duplex bandwidth specified in Kilo, Mega, or Gigabits, or as a percentage of link bandwidth.
385: * Limit~--Sets the maximum bandwidth usage that can not be exceeded.
386: * Guarantee~--Sets the entitled or minimum bandwidth. Bandwidth provided can exceed the guaranteed value.
387: * Priority~--Sets the relative priority of this flow compared to other flows, in a range from 0 to 100.
388: * Hardware~--Utilize NIC hardware classifier, if one is available.
389: * CPU Binding~--Assign one or more processors for the flow’s kernel and interrupt processing. For dedicated use of a set of processors, use {{code}} prset(1m){{/code}} to define a processor set.
390: 
391: * Do I have to set bandwidth limits or guarentees?
392: 
393: No, it is possible to create a flow without limits yet bind it to software or hardware resources.
394: 
395: ===  Miscellaneous
396: 
397: * Which NICs have hardware clasification capabilities?
398: 
399: It is difficult to determine a NIC’s hardware capabilities. Please provide feedback on experiences with specific NICs and the information will be aggregated here. Thanks.
400: 
401:  We are planning to provide an option of dladm which will display these hardware capabilities in a future version of Crossbow.
402: 
403: * How can I tell whether my NIC has hardware receive ring classification capabilities?
404: 
405: At this time you can not tell. There is work underway to add such a capability to the {{code}} dladm(1m) {{/code}} command.
406: 
407: * What do I do if I don’t have enough NICs for all the zones?
408:  One option may be to use VLANs. See [[An example of using IP Instances with VLANs.>>http://blogs.sun.com/stw/entry/using_ip_instances_with_vlans]]
409: // Contributors: Crossbow engineering, Crossbow community at opensolaris.org //
410: // Maintained by: Steffen Weiberle (steffen dot weiberle at sun dot com) //
OpenSolaris

Top Menu

Wiki code for FAQ

Search

Collectives

Community Group

Project

User Group

Subsites

Project crossbow Pages
