Network Virtualization and Resource Control (Crossbow) FAQ

Overview

What is Crossbow?

Crossbow is a set of technologies that provide network virtualization and greatly improves the resource control, performance, and network utilization needed to achieve true OS virtualization, utility computing, and server consolidation.

Crossbow consists of multiple components:

* IP Instances

Crossbow is designed to add network virtualization to Solaris without introducing any performance penalty. Some of the underlying work is delivering better network performance. Receive rings and hardware classification, and multiple MAC address support contribute to better performance and enhance the virtualization provided.

Flow management could introduce some overhead. If the NIC or VNIC is not doing any fanout and B/W control, Crossbow can map a flow directly to a receive (RX) ring and use hardware to classify it. In that case, there is no performance impact. In cases where the NIC or VNIC is already doing bandwidth control or traffic fanout across multiple CPUs, any flow configured on top will have to go through an additional classification layer and there will be a small performance hit.

When will Crossbow be available?

Crossbow is available in OpenSolaris 2009.06, Solaris Express Community Edition (SX-CE) build 105 and later, or prior versions of OpenSolaris that have been updated with the most recent release or development packages.

How do I get Crossbow?

Crossbow is installed with OpenSolaris 2009.06 or an update of the packages of an earlier release of OpenSolaris. Installing SX-CE build 105 or later also includes the features.

IP Instances

What are IP Instances?

IP Instances are separate views of the IP stack, so that visibility and control is limited to the entity (zone) that the instance is assigned to. By default, all of Solaris has one view of IP, and therefor central visibility and control. With zones, the ability to view and control is limited by privileges, and all zones' network traffic decisions are made with a global view by the kernel. When IP instances are used, the view is limited to that information that applies to the instance, not the full kernel. So routing decisions, for example, are made based on the information only in this instance, and does not use any of the additional information that other instances on the same kernel may have. Similarly, control is delegated to this instance, so that a non-global zone can set network parameters such as routes, ndd(1m) values, IP address(es). Snooping of the interface(s) in the IP Instance is also possible. There is no visibility into any of the other IP Instances that may be sharing this Solaris instance and kernel.

Another feature with IP Instances is that traffic between zones must pass the whole path down the stack to the underlying NIC or VNIC. This is the result of the zone's IP not knowing where the destination address is, and outgoing data must thus be put on the wire. If the zone is using a VNIC, whether the traffic stays within the system or exists on a physical netowrk interface depends on whether the destination also using a VNIC sharing the same physical NIC. If a NIC is shared for VNICs, traffic directly between the VNICs will be switched by the VNICs' virtual switch to the destination VNIC, and it will not leave the system.

How do I get them?

IP Instances are in Solaris Nevada build 57 and later.

When will they be available in Solaris 10?

IP Instances are in Solaris 10 8/07 released on 4 September 2007.

Can any NIC be used with IP Instances?

All NICs in OpenSolaris and in Solaris Express Community Edition build 80 or later work with IP Instances. For Solaris 10, only NICs supported by the Generic LAN Driver version 3 (GLDv3) are supported with IP Instances. The way to determine if a NIC is GLDv3, run the dladm(1m) command with the 'show-link' subcommand and look for links that are not of type 'legacy'.

There is one exception. The ce interfaces can also be used now. See Which NICs are known to work with IP Instances? for details, such as Nevada build and Solaris 10 patches required.

This is how non-GLDv3 interfaces will look. (Solaris 10 only)


# dladm show-link
eri0            type: legacy    mtu: 1500       device: eri0
qfe0            type: legacy    mtu: 1500       device: qfe0
qfe1            type: legacy    mtu: 1500       device: qfe1
qfe2            type: legacy    mtu: 1500       device: qfe2
qfe3            type: legacy    mtu: 1500       device: qfe3

And how GLDv3 interfaces look.


# dladm show-link
bge0            type: non-vlan  mtu: 1500       device: bge0
bge1            type: non-vlan  mtu: 1500       device: bge1
bge1001         type: vlan 1    mtu: 1500       device: bge1
bge2001         type: vlan 2    mtu: 1500       device: bge1
bge2            type: non-vlan  mtu: 1500       device: bge2
bge3            type: non-vlan  mtu: 1500       device: bge3
aggr1           type: non-vlan  mtu: 1500       aggregation: key 1

Which NICs are known to work with IP Instances?
- afe (Nevada build 73 and later)
- bge
- ce (Nevada build 80 and later, Solaris 10 with IP Instance patches*)
- dmfe (Nevada build 73 and later)
- e1000g
- eri (Nevada build 73 and later)
- hme (Nevada build 73 and later)
- iprb (Nevada build 73 and later)
- ixgb
- mxfe (Nevada build 73 and later)
- nge
- nxge
- qfe (Nevada build 73 and later)
- rge
- rtls (Nevada build 73 and later)
- xge
- ath (Nevada only)
NOTE: The ce NIC is not a GLDv3 device, but has been made to work with IP Instances. The patches required for Solaris 10 8/07 are:
- 118777-12 and 137042-01 (SPARC)
- 118778-11 and 137043-01 (i386, x86, x64)

Which NICs don't work with IP Instances?
- ce (Nevada build 79 and earlier, and Solaris 10 8/07 or earlier without the patches listed above)
- dnet
- elx
- fjgi
- ge
- ipge
- ixge
- spwr
NOTE: The e1000g driver replaces ipge in Solaris 10 11/06 and later for these NICs:
- Sun PCI-Express Dual Gigabit Ethernet UTP X7280A-2
- Sun PCI-Express Dual Gigabit Ethernet MMF X7281A-2

The Nemo Unification within Project Clearview has delivered a shim that allows non-GLDv3 interfaces to be used with all the Crossbow and IP Instance features. All releases of OpenSolaris and SX-CE since build 83 have this.

(Solaris 10 only) There are two Change Requests to enable IP Instances with the ce driver. See What's Up ce-Doc? for some details. These fixes have been put into OpenSolaris and are available in Nevada build 80 and later, and available for Solaris 10 with the patches listed above.

Can I have more than one IP Instance on a system?

Yes.

How many IP Instances are supported?

There is no set limit on the number of IP Instances. Since each IP Instance needs (for connectivity) at least one data link (NIC, VLAN-tagged NIC, aggregation, or VNIC), the limit is essentially determined by the number of data links available and manageable on the system.

How many IP Instances can a non-global zone have?

A non-global zone can have only one IP Instance. By default, a zone is in the global instance sharing IP with the global zone and all other zones without an exclusive IP Instance. When a zone is configured to have an exclusive IP Instance, its view of IP is now isolated from the rest of the system.

Can an IP Instance be shared between two non-global zones?

No.

Can the global zone see into another zone's IP Instance?

Commands at the IP level such as ifconfig(1m) will only work with the interfaces in the IP Instance from which the command is run. In the global zone, they will only be able to see those interfaces not set as exclusive to a non-global zone.

The snoop(1m) command is able to be used from the global zone even if that interface has been given to a non-global zone with an IP Instance configured. If snoop is run in the global zone and in the zone that has exclusive access to the interface, they will see the same data.

The dladm(1m) command is used from the global zone to manage all devices, links, aggregations, VLANs, and VNICs. At this time, dladm can not be used in a non-global zone, even if it is configured as an exclusive zone.

How can any of the interfaces assigned to a non-global zone be seen?

Using the dladm(1m) command, a privileged user in the global zone can see and control the physical interfaces (NICs, link aggregations (aggr), VLANs, and VNICs).

I have configured a zone with ip-type=exclusive, however I don't see the interface in the zone when running ifconfig.

All interfaces assigned to a non-global zone can be identified by running 'ifconfig -a plumb', followed by 'ifconfig -a'.

non-global zone# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
    inet 127.0.0.1 netmask ff000000
non-global zone# ifconfig -a plumb
non-global zone# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
    inet 127.0.0.1 netmask ff000000
bge1: flags=201000842<BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2
    inet 0.0.0.0 netmask 0
    ether 0:3:ba:e3:42:8c
non-global zone#

How do I configure an IP address for a zone?

If you have, for example, an nge interface, one method is to create the file /etc/hostname.nge0 in the non-global zone.

non-global zone# echo "192.168.1.11/24" > /etc/hostname.nge0
non-globalzone# init 6
...
non-global zone# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
nge0: flags=1000843"<"UP,BROADCAST,RUNNING,MULTICAST,IPv4">" mtu 1500 index 2
        inet 192.168.1.11 netmask ffffff00 broadcast 10.1.14.255
        ether 0:17:31:46:d8:eb
non-global zone#

Generally, you will set up the /etc/hosts file, /etc/defaultrouter if using default routes, /etc/netmasks, /etc/resolv.conf, and the like, as with any stand-alone system. With a shared IP Instance, much of this is managed by the adminstrator(s) in the global zone.

What services can now run in a zone that could not before?
- DHCP client
- DHCP server
- Routing daemon
- IPsec
- IPfilter
- IP Multipathing (IPMP)
- ndd commands
- ifconfig (with set or modify capabilities)

How do I configure the non-global zone to be a DHCP client?

After configuring and installing the zone, copy or create an /etc/sysidcfg file. For example,

global-zone# cat /myzones/dhcpzone/root/etc/sysidcfg
system_locale=C
terminal=xterm
network_interface=primary {
        dhcp
        protocol_ipv6=no
}
security_policy=NONE
name_service=NONE
nfs4_domain=dynamic
timezone=US/Eastern
root_password=""
global-zone# zlogin dhcpzone ifconfig -a
lo0: flags=2001000849"<"UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL">" mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
bge2: flags=201004843"<"UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4,CoS">" mtu 1500 index 2
        inet 10.1.14.161 netmask ffffffc0 broadcast 10.1.14.191
        ether 0:3:ba:e3:42:8d
global-zone#

Or, you can manually bring up the interface with DHCP.


dhcp-zone# ifconfig bge2 plumb
dhcp-zone# ifconfig bge2 auto-dhcp

To have it use DHCP on subsequent boot ups.


dhcp-zone# touch /etc/hostname.bge2
dhcp-zone# touch /etc/dhcp.bge2

What about NFS in a non-global zone?

A non-global zone can still be an NFS client (not of the global zone on the same system), but can not be an NFS server. The in-ability of a non-global zone to be an NFS server is not related to networking, but rather to file system and virtual memory interaction. Whether or not IP Instances, VNICs, or any other Crossbow feature is used does not affect or change this situation.

Is there anything else that does not work?

You can not load private kernel modules in a non-global zone, even if you have your own instance. Also, IPfilter rulesets are controlled from the global zone at this time. A linux branded zone does not work with IP Instances at this time.

Virtual NICs (VNICs)

What is a VNIC?

A VNIC is a virtualized network interface that presents the same media access control (MAC) interface that an actual interface would provide. Multiple VNICs can be configured on top of the same interface, allowing multiple consumers to share that interfaces. If the interface has hardware clasification capabilities, when data arrives on the NIC, the hardware can automatically direct the datagrams to receiver buffers (rings) associated with a specific VNIC. It may be possible to selectively turn interrupts on and off per ring, allowing the host to control the rate of arrival of packets into the system. For hardware that does not have these capabilities, these features are provided via software.

Which NICs support VNICs?

VNICs are support on Generic LAN Driver version 3 (GLDv3) supported interfaces. For a list, see Which NICs are known to work with IP Instances? You can also create a VNIC on top of an aggregation or VLAN that is built using GLDv3 NICs.

How many VNICs can I create per NIC?

There is no limit to the number VNICs per NIC. However, for NICs with hardware classification capabilities, maximum performance is achieved when the number of VNICs does not exceed the number of hardware classifiers on the NIC.

How many VNICs can I create on a system?

There is no maximum number of VNICs on a system (there use to be and it has been removed). As is typically the case, each VNIC will require additional system resources such as CPU. So there will be a practical maximum per system based on the type of system, the type of NIC(s), and traffic patterns.

How is a VNIC managed?

The dladm(1m) command is used to create, modify, and delete VNICs.

How do I dermine whether the NIC has its own set of factory-provided MAC addresses?

Use dladm show-dev -m to show the MAC addresses assigned to this NIC.

How do I use a factory provided MAC address?

To use a factory provided MAC address, run dladm create-vnic -m factory when creating a VNIC.

How do I set my own MAC address?

The MAC address for a VNIC can be set when the VNIC is created with the dladm create-vnic -m MAC-address command in the global zone.

Future???: The MAC address can be modified using the ifconfig(1m) command in a non-global zone. Either operation must be done by a privileged administrator.

Are there restrictions regarding the MAC addresses I can use?

Yes, the MAC address must be a valid MAC address as per IEEE. It can not be a multicast or broadcast address.

Is it necessary to always specify the MAC address of a VNIC?

No, you can choose to use a factory MAC address, have the system generate a random MAC address, or provide one. The default is to use an available factory address, and if none is available, generate a random one.

Can I use IPv6 addresses with a VNIC?

Yes. You can do most of the things that you can do with a physical NIC. Things you can not do with a VNIC include: create a link aggregation, set a frame size larger than the underlink link.

Can different VNICs have different MTUs set?

Yes, but no larger than the MTU allowed by the underlying NIC.

When will VNICs be available in Solaris 10?

At this time there is no specific plan to back port VNICs to Solaris 10. If you have a need for VNIC support in Solaris 10, please add a service record to change request (CR) 6790102.

Flow Management

What is flow management?

Flow management is the ability to manage networking bandwidth resource for a transport, service, or a virtual machine. A service is specified as a combination of transport (e.g. TCP, UDP) and port, while a virtual machine is specified by its mac address or an IP address.

How do I define a flow?

Flows are managed with the flowadm(1m) command.

What parameters control a flow?

Flows are defined as a set of attributes based on Layer 2, 3, and 4 headers which can be used to identify a protocol, service, or a [virtual machine] instance, such as a zone or Xen domain.

Which attributes can I use to define the flow?
- local_mac_addr
- SAP
- local_ip_addr
- transport (TCP, UDP, or SCTP)
- local_port
- remote_port

What parameters can be set on flows?

Flows support the following parameters:

* Bandwidth--Sets the full duplex bandwidth specified in Kilo, Mega, or Gigabits, or as a percentage of link bandwidth.

Limit--Sets the maximum bandwidth usage that can not be exceeded.
Guarantee--Sets the entitled or minimum bandwidth. Bandwidth provided can exceed the guaranteed value.
Priority--Sets the relative priority of this flow compared to other flows, in a range from 0 to 100.
Hardware--Utilize NIC hardware classifier, if one is available.
CPU Binding--Assign one or more processors for the flow's kernel and interrupt processing. For dedicated use of a set of processors, use prset(1m) to define a processor set.

Do I have to set bandwidth limits or guarentees?

No, it is possible to create a flow without limits yet bind it to software or hardware resources.

Miscellaneous

Which NICs have hardware clasification capabilities?

It is difficult to determine a NIC's hardware capabilities. Please provide feedback on experiences with specific NICs and the information will be aggregated here. Thanks.

We are planning to provide an option of dladm which will display these hardware capabilities in a future version of Crossbow.

How can I tell whether my NIC has hardware receive ring classification capabilities?

At this time you can not tell. There is work underway to add such a capability to the dladm(1m) command.

What do I do if I don't have enough NICs for all the zones?
One option may be to use VLANs. See An example of using IP Instances with VLANs.
Contributors: Crossbow engineering, Crossbow community at opensolaris.org
Maintained by: Steffen Weiberle (steffen dot weiberle at sun dot com)

last modified by admin on 2009/10/26 12:13

OpenSolaris

Top Menu

Network Virtualization and Resource Control (Crossbow) FAQ

Table of Contents

Overview

IP Instances

Virtual NICs (VNICs)

Flow Management

Miscellaneous

Overview

IP Instances

Virtual NICs (VNICs)

Flow Management

Miscellaneous

Search

Collectives

Community Group

Project

User Group

Subsites

Project crossbow Pages