Wiki code for DTrace Network Provider

Hide Line numbers
1: =DTrace Network Providers
2: 
3: The following is a design proposal for a collection of DTrace Networking Providers. These providers aim to provide networking observability and troubleshooting information for Solaris users. The first prototype TCP provider was [[demonstrated>>Community Group dtrace.CEC2006]] at CEC 2006.
4: 
5: This document will list the probes that may be made available, their arguments, and link prototypes and examples. Feedback is welcome, please post to [[dtrace-discuss>>http://mail.opensolaris.org/mailman/listinfo/dtrace-discuss]]. NOTE: this is a work in progress, and most of these network providers are not yet available.
6: 
7: {{toc start=2 depth=3 numbered=false scope=page /}}
8: 
9: ====Who is talking to my web server?
10: 
11: {{{
12: # dtrace -n ’tcp:::receive /args[2]->tcp_dport == 80/ {
13:         @pkts[args[1]->ip_daddr] = count();
14: }’
15: dtrace: description ’tcp:::receive’ matched 1 probe
16: ^C
17: 
18:   192.168.1.8                                                       9
19:   fe80::214:4fff:fe3b:76c8                                         12
20:   192.168.1.51                                                     32
21:   10.1.70.16                                                       83
22:   192.168.7.3                                                     121
23:   192.168.101.101                                                 192
24: }}}
25: 
26: ====What ports are people connecting to?
27: 
28: {{{
29: # dtrace -n ’tcp:::accept-established { @[args[2]->tcp_dport] = count(); }’
30: dtrace: description ’tcp:::accept-established’ matched 1 probe
31: ^C
32: 
33:        79                2
34:        22               14
35:        80              327
36: }}}
37: 
38: ==Prototypes
39: 
40: The following prototypes demonstrate older designs than what is described on this main page, however what they demonstrate should still be doable albiet with slightly different probe names and arguments.
41: 
42: * [[**Prototype #4**>>http://wikis.sun.com/display/DTrace/tcp+Provider]], Aug 2008. TCP provider. Similar to the TCP probes from prototype #1. The [[source code>>http://cr.opensolaris.org/~bgregg/tcp-provider-prototype2]] is available.
43: * [[**Prototype #3**>>http://wikis.sun.com/display/DTrace/ip+Provider]], Dec 2007. IP provider (covers IPv4 and IPv6). Similar to prototype #2, but with the probe arguments as described in the Arguments section below.
44: * **Prototype #2**, Jun 2007. IP, IPv4, IPv6 providers.
45: * **Prototype #1**, Oct 2006. TCP, UDP and IP providers. Also see the [[CEC2006 Demo>>Community Group dtrace.CEC2006]].
46: 
47: ==Aims
48: 
49: The main aims of the network providers are:
50: 
51: * Support for common network protocols: TCP, UDP, IP.
52: * Everyday observability: connections/packets/bytes by address/port.
53: * Performance observability: throughputs, network latencies.
54: * Socket level tracing: associate events to PIDs.
55: * Further troubleshooting support: TCP flag analysis, header info.
56: 
57: Future enhancements and additions to the providers include:
58: 
59: * Other protocol support: ICMP, SCTP, ...
60: * Kernel code path latency probes.
61: 
62: ==News
63: 
64: The IP provider was integrated into Solaris Nevada build 93, and OpenSolaris. The official documentation is here: [[DTrace IP Provider>>http://wikis.sun.com/display/DTrace/ip+Provider]].
65: 
66: The TCP provider is now being prepared for integration, and prototype documentation is available here: [[DTrace TCP Provider>>http://wikis.sun.com/display/DTrace/tcp+Provider]].
67: 
68: ==Providers
69: 
70: The Network provider will be a collection of several DTrace providers, such as ipv4, ipv6 and tcp (there is no monolithic net provider). The proposed providers are,
71: 
72: |=Provider|=Description
73: |gld|This traces the generic LAN device layer, and shows link layer activity such as Ethernet frames. The probes allow frame by frame tracing.
74: |arp|This traces ARP and RARP packets.
75: |icmp|This traces ICMP packets, and provides the type and code from the ICMP header.
76: |ip|This traces high level IP details for IPv4 and IPv6.
77: |tcp|This traces the TCP layer, and shows what TCP activity is occuring.
78: |udp|This traces UDP events.
79: |sctp|This traces SCTP events.
80: |socket|This traces the socket layer, close to the application. These probes fire in the same context as the corresponding process, and show data sent to or received from sockets.
81: 
82: Other providers will be added as required.
83: 
84: ==Probes
85: 
86: The following is the master plan for the net providers - what probes could be exported, what their arguments could be, and how they fit together. See the prototype pages for implementation details.
87: 
88: |=probes|=args[0]|=args[1]|=args[2]|=args[3]|=args[4]|=args[5]
89: |gld:::send, gld:::receive|pktinfo_t *|NULL|ipinfo_t|illinfo_t *|etherinfo_t *| 
90: |ip:::send, ip:::receive|pktinfo_t *|csinfo_t|ipinfo_t *|illinfo_t *|ipv4info_t *|ipv6info_t *
91: |tcp:::send, tcp:::receive|pktinfo_t *|csinfo_t *|ipinfo_t *|tcpsinfo_t *|tcpinfo_t *| 
92: |tcp:::accept-*, tcp:::connect-*|pktinfo_t *|csinfo_t *|ipinfo_t *|tcpsinfo_t *|tcpinfo_t *| 
93: |tcp:::state-change|NULL|csinfo_t *|tcpsinfo_t *|tcpnsinfo_t *| | 
94: |tcpf:::send, tcpf:::receive|pktinfo_t *|csinfo_t *|ipinfo_t *|tcpsinfo_t *|tcpfinfo_t *| 
95: |udp:::send, udp:::receive|pktinfo_t *|csinfo_t *|ipinfo_t *|udpinfo_t *| | 
96: |udp:::stream-*|pktinfo_t *| | | | | 
97: |icmp:::send, icmp:::receive|pktinfo_t *|NULL|ipinfo_t *|illinfo_t *|icmpinfo_t *| 
98: 
99: Other probes will be added as required.
100: 
101: ==Arguments
102: 
103: The role of the arguments listed above is as follows:
104: 
105: * pktinfo_t * - packet info; After task 7 in the Customer Provider Plan in the following section, this will provide packet IDs to allow a packet to be followed as it passes through the network stack. For now, it will use mblk pointers as packet IDs.
106: * csinfo_t * - connection state info; After task 13 in the Customer Provider Plan in the following section, this will provide connection IDs to assist associating packets with user-level process IDs. For now, it will use conn_t pointers as packet IDs.
107: * ipinfo_t * - IP info; basic IP info that is available throughout the stack.
108: * illinfo_t * - IP lower level info; details about the network interface.
109: * etherinfo_t * - Ethernet header info.
110: * ipv4info_t * - IPv4 header info.
111: * ipv6info_t * - IPv6 header info.
112: * tcpinfo_t * - TCP header info.
113: * tcpfuseinfo_t * - TCP fusion details.
114: * udpinfo_t * - UDP header info.
115: * icmpinfo_t * - ICMP header info.
116: 
117: Other arguments will be added as required.
118: 
119: ==Plan
120: 
121: Network stack instrumentation with DTrace will be achieved through two projects, one to create customer-orientated providers, the other for engineer-orintated providers:
122: 
123: * public/stable/customer providers for generic observability
124: ip:::, tcp:::, udp:::, ...
125: * private/unstable/engineer interface for code path observability
126: sdt:::ip*
127: 
128: ===Customer Provider Plan
129: 
130: The following are rough steps for creating a suite of customer-orientated network providers. Each step may be integrated separately.
131: 
132: {{{
133: 1. IP Provider
134:         ip:::send
135:         ip:::receive
136: 
137: 2. TCP Provider
138:         tcp:::send
139:         tcp:::receive
140:         tcp:::accept-*
141:         tcp:::connect-*
142:         tcp:::state-*
143:         tcp:::retransmit
144:         tcp:::timeout-*
145:         tcpf:::send
146:         tcpf:::receive
147: 
148: 3. UDP Provider
149:         udp:::send
150:         udp:::receive
151:         udp:::stream-*
152: 
153: 4. IP Provider
154:         ip:::drop-in
155:         ip:::drop-out
156:         (usr/src/uts/common/inet/ip/ipdrop.c)
157: 
158: 5. IP Provider
159:         ip:::request
160:         ip:::deliver
161: 
162: 6. IP packet ids
163:         ip:::packet-free
164:         perhaps a new mblk_t member: (uint64_t)mblk_t->b_id;
165:         future, any suitible uint64_t: (uint64_t)nifty_t->n_id;
166: 
167: 7. Socket Provider
168:         socket:::send
169:         socket:::receive
170: 
171: 8. ICMP Provider
172:         icmp:::send
173:         icmp:::receive
174: 
175: 9. ARP Provider
176:         arp:::send
177:         arp:::receive
178: 
179: 10. SCTP Provider
180:         sctp:::send
181:         sctp:::receive
182: 
183: 11. GLD Provider
184:         gld:::send
185:         gld:::receive
186: 
187: 12. TCP/UDP connection ids
188:         /connfs
189: }}}
190: 
191: Dropped from the list:
192: 
193: {{{
194: 13. DTrace Enhancements
195:         write() action
196:         msgdcopy() action
197: }}}
198: 
199: Task 13 would allow a DTrace script to output an RFC 1761 file for reading using snoop or wireshark. A prototype was written, but work has been halted in favour of the other tasks in the plan. Snoop is more suitable for capturing packet data on high speed networks.
200: 
201: ===Engineer Provider Plan
202: 
203: Many probes have already been integrated into Solaris Nevada for detailed observability of network stack behaviour:
204: 
205: {{{
206: dtrace -ln ’sdt:::ip*’ | awk ’BEGIN { OFS=":::" } { print $2,$5 }’
207: PROVIDER:::NAME
208: sdt:::ip-ire-del
209: sdt:::ip-ire-del-origin
210: sdt:::ip6-physical-out-end
211: sdt:::ip6-physical-out-end
212: sdt:::ip6-physical-out-start
213: sdt:::ip6-loopback-in-end
214: sdt:::ip6-loopback-in-end
215: sdt:::ip6-loopback-in-start
216: sdt:::ip6-loopback-in-start
217: sdt:::ip6-forwarding-end
218: sdt:::ip6-forwarding-start
219: sdt:::ip6-physical-in-end
220: sdt:::ip6-physical-in-start
221: sdt:::ip-xmit-incomplete
222: sdt:::ip-xmit-v4
223: sdt:::ip-wput-ipsec-bail
224: sdt:::ip6-loopback-out-end
225: sdt:::ip6-loopback-out-end
226: sdt:::ip6-loopback-out-end
227: sdt:::ip6-loopback-out-start
228: sdt:::ip6-loopback-out-start
229: sdt:::ip6-loopback-out-start
230: sdt:::ip4-loopback-in-end
231: sdt:::ip4-loopback-in-end
232: sdt:::ip4-loopback-in-start
233: sdt:::ip4-loopback-in-start
234: sdt:::ip4-loopback-out_end
235: sdt:::ip-xmit-1
236: sdt:::ip-xmit-2
237: sdt:::ip4-loopback-out-end
238: sdt:::ip4-loopback-out-end
239: sdt:::ip4-loopback-out-end
240: sdt:::ip4-loopback-out-end
241: sdt:::ip4-loopback-out-start
242: sdt:::ip4-loopback-out-start
243: sdt:::ip4-loopback-out-start
244: sdt:::ip4-loopback-out-start
245: sdt:::ip4-dhcpinit-pkt
246: sdt:::ip4-physical-in-end
247: sdt:::ip4-physical-in-start
248: sdt:::ip4-physical-out-end
249: sdt:::ip4-physical-out-end
250: sdt:::ip4-physical-out-end
251: sdt:::ip4-physical-out-end
252: sdt:::ip4-physical-out-end
253: sdt:::ip4-physical-out-end
254: sdt:::ip4-physical-out-end
255: sdt:::ip4-physical-out-end
256: sdt:::ip4-physical-out-end
257: sdt:::ip4-physical-out-start
258: sdt:::ip4-physical-out-start
259: sdt:::ip4-physical-out-start
260: sdt:::ip4-physical-out-start
261: sdt:::ip4-physical-out-start
262: sdt:::ip4-physical-out-start
263: sdt:::ip4-physical-out-start
264: sdt:::ip4-physical-out-start
265: sdt:::ip4-physical-out-start
266: sdt:::ip4-forwarding-end
267: sdt:::ip4-forwarding-end
268: sdt:::ip4-forwarding-start
269: sdt:::ip4-forwarding-start
270: sdt:::ip-newroute-drop
271: sdt:::ip6-physical-out_start
272: }}}
273: 
274: In particular, the probes added as part of ipf (eg, sdt:::ip4-physical-out-start) form a good model to build upon.
275: 
276: ==Testing
277: 
278: Ongoing testing of the network providers will be needed for maintaining their accuracy. It is expected to be achieved through two forms of testing:
279: 
280: 1. **Customer testing.** The SUNWdtrt (DTrace Test Suite) package will include basic tests that can be initiated from a single host, as with the other tests in SUNWdtrt. These tests will generate simple network traffic which will be traced, confirming that the probes and arguments work as expected. These tests are expected to be frequently executed by engineers as part of existing ongoing testing of SUNWdtrt, and by customers when troubleshooting DTrace. There are limitations with single host testing, and some code paths such as those that require special hardware (crypto accelerator cards, NICs that support LSO, etc.) will usually not be tested.
281: 1. **Engineer testing.** Using multiple hosts with a variety of different hardware, tests can be executed by hand by generating traffic, tracing details, and confirming that all values are as expected. This will happen during development of each task from the plan and if/whenever bugs are lodged. A test team at Sun with access to a test farm could take this onboard as an ongoing task.
282: 
283: ==Solutions
284: 
285: Some of the network obserability problems that this provide will solve include:
286: 
287: * Who is connecting to my web server? (TCP accepts by IP address)
288: * Which services are busy? (bytes by destination port)
289: * Which hosts are responsible for network load? (bytes by IP address)
290: * Are hackers/crackers port scanning my server? (TCP flag matching by IP address)
291: * Why are outbound connections slow? (Connect latency, 1st byte latency, throughput latency)
292: 
293: ==Comments
294: 
295: This design has been publically open for comments for several months, and discussed a number of times on the [[dtrace-discuss>>http://mail.opensolaris.org/mailman/listinfo/dtrace-discuss]] mailing list. Time is now running out before these probes will begin to be integrated.
296: 
297: Updated: 27-Aug-2008
OpenSolaris

Top Menu

Wiki code for DTrace Network Provider

Search

Collectives

Community Group

Project

User Group

Subsites

Community Group dtrace Pages
