DTrace Network Provider

DTrace Network Providers

The following is a design proposal for a collection of DTrace Networking Providers. These providers aim to provide networking observability and troubleshooting information for Solaris users. The first prototype TCP provider was demonstrated at CEC 2006.

This document will list the probes that may be made available, their arguments, and link prototypes and examples. Feedback is welcome, please post to dtrace-discuss. NOTE: this is a work in progress, and most of these network providers are not yet available.

• Prototypes
• Aims
• News
• Providers
• Probes
• Arguments
• Plan
  • Customer Provider Plan
  • Engineer Provider Plan
• Testing
• Solutions
• Comments

Who is talking to my web server?

# dtrace -n 'tcp:::receive /args[2]->tcp_dport == 80/ {
        @pkts[args[1]->ip_daddr] = count();
}'
dtrace: description 'tcp:::receive' matched 1 probe
^C

  192.168.1.8                                                       9
  fe80::214:4fff:fe3b:76c8                                         12
  192.168.1.51                                                     32
  10.1.70.16                                                       83
  192.168.7.3                                                     121
  192.168.101.101                                                 192

What ports are people connecting to?

# dtrace -n 'tcp:::accept-established { @[args[2]->tcp_dport] = count(); }'
dtrace: description 'tcp:::accept-established' matched 1 probe
^C

       79                2
       22               14
       80              327

Prototypes

The following prototypes demonstrate older designs than what is described on this main page, however what they demonstrate should still be doable albiet with slightly different probe names and arguments.

• Prototype #4, Aug 2008. TCP provider. Similar to the TCP probes from prototype #1. The source code is available.
• Prototype #3, Dec 2007. IP provider (covers IPv4 and IPv6). Similar to prototype #2, but with the probe arguments as described in the Arguments section below.
• Prototype #2, Jun 2007. IP, IPv4, IPv6 providers.
• Prototype #1, Oct 2006. TCP, UDP and IP providers. Also see the CEC2006 Demo.

Aims

The main aims of the network providers are:

• Support for common network protocols: TCP, UDP, IP.
• Everyday observability: connections/packets/bytes by address/port.
• Performance observability: throughputs, network latencies.
• Socket level tracing: associate events to PIDs.
• Further troubleshooting support: TCP flag analysis, header info.

Future enhancements and additions to the providers include:

• Other protocol support: ICMP, SCTP, ...
• Kernel code path latency probes.

News

The IP provider was integrated into Solaris Nevada build 93, and OpenSolaris. The official documentation is here: DTrace IP Provider.

The TCP provider is now being prepared for integration, and prototype documentation is available here: DTrace TCP Provider.

Providers

The Network provider will be a collection of several DTrace providers, such as ipv4, ipv6 and tcp (there is no monolithic net provider). The proposed providers are,

Other providers will be added as required.

Probes

The following is the master plan for the net providers - what probes could be exported, what their arguments could be, and how they fit together. See the prototype pages for implementation details.

Other probes will be added as required.

Arguments

The role of the arguments listed above is as follows:

• pktinfo_t * - packet info; After task 7 in the Customer Provider Plan in the following section, this will provide packet IDs to allow a packet to be followed as it passes through the network stack. For now, it will use mblk pointers as packet IDs.
• csinfo_t * - connection state info; After task 13 in the Customer Provider Plan in the following section, this will provide connection IDs to assist associating packets with user-level process IDs. For now, it will use conn_t pointers as packet IDs.
• ipinfo_t * - IP info; basic IP info that is available throughout the stack.
• illinfo_t * - IP lower level info; details about the network interface.
• etherinfo_t * - Ethernet header info.
• ipv4info_t * - IPv4 header info.
• ipv6info_t * - IPv6 header info.
• tcpinfo_t * - TCP header info.
• tcpfuseinfo_t * - TCP fusion details.
• udpinfo_t * - UDP header info.
• icmpinfo_t * - ICMP header info.

Other arguments will be added as required.

Plan

Network stack instrumentation with DTrace will be achieved through two projects, one to create customer-orientated providers, the other for engineer-orintated providers:

• public/stable/customer providers for generic observability
  ip:::, tcp:::, udp:::, ...
• private/unstable/engineer interface for code path observability
  sdt:::ip*

Customer Provider Plan

The following are rough steps for creating a suite of customer-orientated network providers. Each step may be integrated separately.

1. IP Provider
        ip:::send
        ip:::receive

2. TCP Provider
        tcp:::send
        tcp:::receive
        tcp:::accept-*
        tcp:::connect-*
        tcp:::state-*
        tcp:::retransmit
        tcp:::timeout-*
        tcpf:::send
        tcpf:::receive

3. UDP Provider
        udp:::send
        udp:::receive
        udp:::stream-*

4. IP Provider
        ip:::drop-in
        ip:::drop-out
        (usr/src/uts/common/inet/ip/ipdrop.c)

5. IP Provider
        ip:::request
        ip:::deliver

6. IP packet ids
        ip:::packet-free
        perhaps a new mblk_t member: (uint64_t)mblk_t->b_id;
        future, any suitible uint64_t: (uint64_t)nifty_t->n_id;

7. Socket Provider
        socket:::send
        socket:::receive

8. ICMP Provider
        icmp:::send
        icmp:::receive

9. ARP Provider
        arp:::send
        arp:::receive

10. SCTP Provider
        sctp:::send
        sctp:::receive

11. GLD Provider
        gld:::send
        gld:::receive

12. TCP/UDP connection ids
        /connfs

Dropped from the list:

13. DTrace Enhancements
        write() action
        msgdcopy() action

Task 13 would allow a DTrace script to output an RFC 1761 file for reading using snoop or wireshark. A prototype was written, but work has been halted in favour of the other tasks in the plan. Snoop is more suitable for capturing packet data on high speed networks.

Engineer Provider Plan

Many probes have already been integrated into Solaris Nevada for detailed observability of network stack behaviour:

dtrace -ln 'sdt:::ip*' | awk 'BEGIN { OFS=":::" } { print $2,$5 }'
PROVIDER:::NAME
sdt:::ip-ire-del
sdt:::ip-ire-del-origin
sdt:::ip6-physical-out-end
sdt:::ip6-physical-out-end
sdt:::ip6-physical-out-start
sdt:::ip6-loopback-in-end
sdt:::ip6-loopback-in-end
sdt:::ip6-loopback-in-start
sdt:::ip6-loopback-in-start
sdt:::ip6-forwarding-end
sdt:::ip6-forwarding-start
sdt:::ip6-physical-in-end
sdt:::ip6-physical-in-start
sdt:::ip-xmit-incomplete
sdt:::ip-xmit-v4
sdt:::ip-wput-ipsec-bail
sdt:::ip6-loopback-out-end
sdt:::ip6-loopback-out-end
sdt:::ip6-loopback-out-end
sdt:::ip6-loopback-out-start
sdt:::ip6-loopback-out-start
sdt:::ip6-loopback-out-start
sdt:::ip4-loopback-in-end
sdt:::ip4-loopback-in-end
sdt:::ip4-loopback-in-start
sdt:::ip4-loopback-in-start
sdt:::ip4-loopback-out_end
sdt:::ip-xmit-1
sdt:::ip-xmit-2
sdt:::ip4-loopback-out-end
sdt:::ip4-loopback-out-end
sdt:::ip4-loopback-out-end
sdt:::ip4-loopback-out-end
sdt:::ip4-loopback-out-start
sdt:::ip4-loopback-out-start
sdt:::ip4-loopback-out-start
sdt:::ip4-loopback-out-start
sdt:::ip4-dhcpinit-pkt
sdt:::ip4-physical-in-end
sdt:::ip4-physical-in-start
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-end
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-physical-out-start
sdt:::ip4-forwarding-end
sdt:::ip4-forwarding-end
sdt:::ip4-forwarding-start
sdt:::ip4-forwarding-start
sdt:::ip-newroute-drop
sdt:::ip6-physical-out_start

In particular, the probes added as part of ipf (eg, sdt:::ip4-physical-out-start) form a good model to build upon.

Testing

Ongoing testing of the network providers will be needed for maintaining their accuracy. It is expected to be achieved through two forms of testing:

1. Customer testing. The SUNWdtrt (DTrace Test Suite) package will include basic tests that can be initiated from a single host, as with the other tests in SUNWdtrt. These tests will generate simple network traffic which will be traced, confirming that the probes and arguments work as expected. These tests are expected to be frequently executed by engineers as part of existing ongoing testing of SUNWdtrt, and by customers when troubleshooting DTrace. There are limitations with single host testing, and some code paths such as those that require special hardware (crypto accelerator cards, NICs that support LSO, etc.) will usually not be tested.
2. Engineer testing. Using multiple hosts with a variety of different hardware, tests can be executed by hand by generating traffic, tracing details, and confirming that all values are as expected. This will happen during development of each task from the plan and if/whenever bugs are lodged. A test team at Sun with access to a test farm could take this onboard as an ongoing task.

Solutions

Some of the network obserability problems that this provide will solve include:

• Who is connecting to my web server? (TCP accepts by IP address)
• Which services are busy? (bytes by destination port)
• Which hosts are responsible for network load? (bytes by IP address)
• Are hackers/crackers port scanning my server? (TCP flag matching by IP address)
• Why are outbound connections slow? (Connect latency, 1st byte latency, throughput latency)

Comments

This design has been publically open for comments for several months, and discussed a number of times on the dtrace-discuss mailing list. Time is now running out before these probes will begin to be integrated.

Updated: 27-Aug-2008