Export

View

DTrace Networking Cookbook

Dtrace Networking Cookbook

From CTG

Author: Sowmini Varadhan

Background Information

Dtrace is a handy tool for debugging some types of networking problems esp. in the kernel, and this cookbook will attempt to provide a quick tour of dtrace, what it can/can't do, and a collection of handy networking specific scripts of varying complexity. The objective is to provide a repository of scripts that can be borrowed from, or built upon, by someone in the middle of a nasty network bug, who's thinking "Gee, I wish I could find out what's happening here but I don't have the time to take a Dtrace class!"

Note that the scripts included here are mostly intended to be used as sources for cut/paste templates for solving other problems, rather than as complete "cure-all" examples of their own.

Contributions of all shapes, sizes and complexity to the repository are welcome! Please send mail to Sowmini with any input!

Here are some useful links to other documentation:

Dtrace for Networking Tutorial

General Start

Guide

How do I use DTrace? You will probably either use DTrace scripts or write DTrace scripts:

Use DTrace scripts

Download the DTraceToolkit
Download Scripts from this website or Other websites.
Check out the DTrace OneLiners.
Use scripts found in /usr/demo/dtrace, or in the DTrace Guide
Seach the BigAdmin Forum for useful script or elsewhere on the Internet.

Write DTrace Scripts

Some people will write their own customised DTrace scripts to troubleshoot faults or solve performance issues.

It helps to know C
It helps to know a little about the kernel
Nearly anything is possible

To get started writing your own scripts:

Read Chapter 1 (at least) of the DTrace Guide
Check out the DTrace OneLiners.
Read through the DTrace Examples.
Study scripts.
See Other websites for scripts.
Download the DTraceToolkit
Participate in the BigAdmin Forum

Note: Many of the DTrace scripts on this site have been wrapped in the Bourne shell or Perl to provide command line options. In the future, DTrace may may be directly capable of optional arguments. In the meantime this code has been written in such a way that future conversion to pure DTrace is easy.

Networking Dtrace Scripts

src.tar.gz

Several dtrace one-liners are available at usres.tpg.com.au

In addition a tarball of Dscripts of varying complexity (detailed descriptions follow) may be found here

Some Simple Dscripts

bge_send.d - count packets sent in bge_send, and the fails. Can easily be modified to count packets of a particular type (like ip proto etc.)

arp_lookup.d - print out ipv4 address that we are trying to lookup in arp.

arp_snoop.d prints stack trace and packet layout (in snoop-like format) of packets intercepted in ar_rput().

break_in_kmdb.html - demonstrates how to get dscripts to stop in kmdb.

ctx.d - watch for context switching during sendto() calls using pswitch probe.

udp.d - use UDP MIB provider to show udp stats once per second.

ip_reass.d - track fragmented ip packets; has a user-friendly output of 32 bit ip addresses..

unknown_proto.d is a dtrace script to print out the identity of "unknown IP protocol" packets -- the things bumping up the ipInUnknownProtos MIB/netstat counter.

Dscripts for Timing Drivers, Functions

e1000g.d - count e1000g driver calls, time the expensive CleanTxInterrupts() function; Demonstrates use of Aggregations ("@").

ce.d.html - measure the service time in the arp module with ce driver.

jitter.d - Measure time spent in sendto, recvfrom, looking for "jitter", or long pauses; jitter2.d refines jitter.d by using speculate()

trace_nce_res_mp.d is a simple template that can be customized to help pick out a particular packet at critical functions (dblk_free, in this example) in the stack.

Dscripts for kernel profiling

prof.d - Profile the kernel looking for high system time; demonstrates how to use the profile provider with aggregations. prof2.d and prof3.d are refinements of prof.d..

Profiling tools built over dtrace

The Sun Studio Compiler also supports the er_kernel tool for profiling the Solaris kernel. er_kernel can collect data using either of two drivers: DTrace, for Solaris 10, which will be externally available, and an no-ship loadable driver, kpcsdrv, available for Solaris 8, 9, or 10. The DTrace driver is the default; the kpcsdrv driver is accessed when the first argument to er_kernel is -X.

A wealth of information about this tool is available at the following locations:

And More Elaborate Ones....

There are some examples of scripts to watch connections at the dtrace discussions (http://www.open solaris.org/os/community/dtrace/scripts/) site.

zerocopy.d (http://npt.eng/dtrace/zerocopy.d.html//)** - monitors

zerocopy.d - monitors TCP's ZeroCopy feature, shows what happens at different components, e.g. STREAM head, TCP, IP(ill), mblk, etc.

Other Dtrace Script Listings

Richard McDougall's Dtrace Scripts

Limitations of Dtrace for Networking

Dtrace is not a good tool for post-mortem analysis. For example, one problem frequently encountered in Networking is the "where did my packet get queued up?" problem posed in this customer's question. Another class of problems encountered is the analysis of the sequence of events that led to a memory leak or reference-count leak after the leak has occurred.

Contributors

Jim Carlson, Jim Fiori, Darrin Johnson, Alex Peng, Sowmini Varadhan

Retrieved from "http://ctg.central.sun.com/wiki/index.php/Dtrace_Networking_Cookbook"
php/Main_Page

Search

Tags:

Created by admin on 2009/10/26 12:08

Last modified by admin on 2009/10/26 12:08