Wiki code for CPU Caps PSARC case draft

Hide Line numbers
1: = CPU Caps
2: 
3: **Alexander Kolbasov (alexander.kolbasov@sun.com) **
4: 
5: [[PDF Version>>Project rm.caps_psarc.pdf]]
6: 
7: =   1 Summary
8: 
9:  CPU caps provide absolute fine-grained limits on the amount of CPU resources that can be consumed by a project or a zone. It is provided as an extension to [[resource_controls(5)>>http://docs.sun.com/app/docs/doc/816-5175/6mbba7f37?a=view]] . This document provides a short description specifically for PSARC. A full description is available in the design document [[[1>>#bib:caps:design]]] and the implementation is discussed in the implementation guide [[[2>>#bib:caps:impl]]]. The on-line version of this document is available in [[[5>>#caps_psarc]]].
10: 
11:  CPU caps extend existing facilities that control CPU usage on the system - processor binding, processor sets and the Fair Share Scheduler. Unlike processor binding and processor sets, CPU caps allow applications to run on any CPUs while limiting their CPU usage and provide fine-grained (specified in fractions of a CPU) limit. In contrast to the [[FSS(7)>>http://docs.sun.com/app/docs/doc/816-5177/6mbbc4g5u?a=view]] , CPU caps provide absolute usage limit that does not depend on other applications running in the system. CPU caps can be used together with CPU binding, processor sets and FSS. When used in conjunction with processor sets, CPU caps limit CPU usage within a set. When used with [[FSS(7)>>http://docs.sun.com/app/docs/doc/816-5177/6mbbc4g5u?a=view]] , CPU resources defined by caps are further subdivided using [[FSS(7)>>http://docs.sun.com/app/docs/doc/816-5177/6mbbc4g5u?a=view]] shares.
12: 
13:  CPU caps provide several advantages over the mechanisms described above:
14: 
15: * Allow customers more flexibility in managing CPU resources than processor sets.
16: * Easier to manage and use than shares.
17: * Provide simpler semantics compared to CPU shares.
18: 
19:  The CPU caps are administered using the [[resource_controls(5)>>http://docs.sun.com/app/docs/doc/816-5175/6mbba7f37?a=view]] framework or the [[zonecfg(1M)>>http://docs.sun.com/app/docs/doc/816-5166/6mbb1kqm2?a=view]] command (see section [[3>>#sec:usage]]). Two new resource controls are introduced to set per-project and per-zone CPU caps (see section [[6.1>>#rctl:extensions]]).
20: 
21:  System administrators can use kstats to observe impact of CPU caps at the zone or project level (see [[5.1>>#observability:kstat]]). The DTrace sched provider is extended with new probes which can be used for per-process or per-thread analysis (see [[5.3>>#dtrace]]).
22: 
23:  All extensions described here are //Committed// except for the kstats which are //Uncommitted//. Patch binding is requested for all.
24: 
25: =  2 Motivation
26: 
27:  Customers provided the following major reasons for wanting CPU caps:
28: 
29: * Selling CPU resources.
30:  Customers selling their CPU resources want to provide usage limits based on the amount of CPU resources purchased.
31: * Managing expectations.
32:  Customers want the clients buying CPU resources to have the same experience in terms of their application performance independent of the machine load. In particular, they do not want clients running on an idle system experience better performance than when running in parallel with other clients applications. Note that such behavior can not be achieved with the FSS since it does not limit CPU resources when idle CPUs are present.
33: * Over-subscription
34:  Some customers want to use hard limits as a mechanism to over-subscribe users. They may sell more CPU resource than what is available and they will be able to provide the level of service unless they run out of CPU cycles in which case there will be some decline in performance.
35: 
36:  In some form, such mechanism for limiting CPU resources is provided by other operating systems. It became one of many check-box items which customers look at when comparing resource management solutions from Sun and other vendors.
37: 
38: ==  2.1 Related CPU resource management tools
39: 
40:  Solaris provides different mechanisms which can be used to control CPU usage of applications to some extent:
41: 
42: * Processor binding (see [[pbind(1M)>>http://docs.sun.com/app/docs/doc/816-5166/6mbb1kqbg?a=view]] ) provides a way to limit process or a set of processes to a single CPU. It allows other unbound threads to run on the same CPU and compete for CPU cycles with bound threads.
43: * Processor sets (see [[psrset(1M)>>http://docs.sun.com/app/docs/doc/816-5166/6mbb1kqd0?a=view]] ) provide a way to limit process execution to a set of CPUs. It also prohibits threads not belonging to the set from running within the set.
44: * Resource pools (see [[pooladm(1M)>>http://docs.sun.com/app/docs/doc/816-5166/6mbb1kqbv?a=view]] ) integrate processor sets with Zones and Projects.
45: * Fair Share Scheduler (see [[FSS(7)>>http://docs.sun.com/app/docs/doc/816-5177/6mbbc4g5u?a=view]] ) provides a mechanism to share available CPUs within given proportions (shares). The //Fair Share Scheduler// only controls CPU usage relative to CPU usage of other applications. CPU shares do not limit CPU usage when there are idle CPU resources.
46: 
47:  CPU binding and processor sets provide //hard// limits for applications since they only run on a bound CPU or within a set, but the minimum granularity for both is a single CPU. The [[FSS(7)>>http://docs.sun.com/app/docs/doc/816-5177/6mbbc4g5u?a=view]] provides //soft// limits which are enforced only when there is enough competition for CPU resources and is specified in units expressing relationships of processes to each other.
48: 
49: =   3 Administrative Interface
50: 
51:  CPU caps are represented by two new resource controls. The cap value is specified in units of one per cent of a CPU regardless of how many CPUs are available or their characteristics. The value should be greater than zero.
52: 
53: ==   3.1 Project CPU caps
54: 
55:  A project CPU cap is represented by the project.cpu-cap resource control. It is associated with the //privileged level// and can be modified only by privileged (superuser) callers (see [[resource_controls(5)>>http://docs.sun.com/app/docs/doc/816-5175/6mbba7f37?a=view]] and [[prctl(1)>>http://docs.sun.com/app/docs/doc/816-5165/6mbb0m9p6?a=view]] for the description of //privileged level//).
56: 
57:  The project.cpu-cap resource limit can be set statically for projects in [[project(4)>>http://docs.sun.com/app/docs/doc/816-5174/6mbb98uiu?a=view]] file. For example, the following line in [[project(4)>>http://docs.sun.com/app/docs/doc/816-5174/6mbb98uiu?a=view]] file sets persistent CPU cap of 6 CPUs for user akolb:
58: 
59: {{{
60: 
61: user.akolb:1234::::project.cpu-cap=(privileged,600,deny)
62: 
63: }}}
64: 
65:  Project CPU cap can be also dynamically modified or removed on a running system using [[prctl(1)>>http://docs.sun.com/app/docs/doc/816-5165/6mbb0m9p6?a=view]] utility. For example, the following command modifies the CPU cap to limit user akolb to 3 CPUs:
66: 
67: {{{
68: 
69: $ prctl -r -t privileged -n project.cpu-cap -v 300 -i project user.akolb
70: 
71: }}}
72: 
73: ==   3.2 Zone CPU caps
74: 
75:  A zone CPU caps is represented by the zone.cpu-cap resource control. Similar to the project cap it is associated with the //privileged level// and can be modified only by privileged (superuser) callers.
76: 
77:  The zone.cpu-cap resource can be set for a zone using [[zonecfg(1M)>>http://docs.sun.com/app/docs/doc/816-5166/6mbb1kqm2?a=view]] command. The following example configures CPU cap for a zone to 3 CPUs:
78: 
79: {{{
80: 
81: zonecfg:myzone> add rctl
82: zonecfg:myzone:rctl> set name=zone.cpu-cap
83: zonecfg:myzone:rctl> add value (priv=privileged,limit=300,action=deny)
84: zonecfg:myzone:rctl> end
85: 
86: }}}
87: 
88: ===  [[zonecfg(1M)>>http://docs.sun.com/app/docs/doc/816-5166/6mbb1kqm2?a=view]] extensions for CPU caps
89: 
90:  The [[zonecfg(1M)>>http://docs.sun.com/app/docs/doc/816-5166/6mbb1kqm2?a=view]] is extended with a new resource called capped-cpu, as described in PSARC/2006/496 [[[3>>#PSARC2006496]]]. The resource value, called ncpus, maps to the zone.cpu-cap rctl.
91: 
92:  The following example sets zone CPU cap using the capped-cpu resource:
93: 
94: {{{
95: 
96:   zonecfg:myzone> add capped-cpu
97:   zonecfg:myzone>capped-cpu> set ncpus=3
98:   zonecfg:myzone>capped-cpu>capped-cpu> end
99: 
100: }}}
101: 
102:  The PSARC/2006/496 [[[3>>#PSARC2006496]]] describes the capped-cpu resource in the following way:
103: 
104: >  The capped-cpu resource has a single ncpus property which is a positive decimal with two digits to the right of the decimal. This property is implemented as a special case of the zonecfg cpu-cap alias. The special case handling of this property normalizes the value so that it corresponds to units of cpus and is similar to the ncpus property under the dedicated-cpu resource group. Unlike dedicated-cpu it will not accept a range and it will accept a decimal number. For example, when using ncpus in the dedicated-cpu resource group, a value of 1 means one dedicated cpu. When using ncpus in the capped-cpu resource group, a value of 1 means 100% of a cpu as the zone.cpu-cap setting. A value of 1.25 means 125%, since 100% corresponds to one full cpu on the system when using cpu caps. The intention here is to align the ncpus units as closely as possible in these two cases (dedicated-cpu vs. capped-cpu), given the limitations and capabilities of the two underlying mechanisms (pset vs. rctl).
105: 
106:  See PSARC/2006/496 [[[3>>#PSARC2006496]]] for a description of the dedicated-cpu resource and ncpu property.
107: 
108:  Here we commit to the capped-cpu resource.
109: 
110: =  4 Implementation Overview
111: 
112:  For all threads running in capped projects or zones, the system keeps track of their CPU usage over short periods of time. When CPU usage of projects or zones reaches specified caps, threads in them do not get scheduled and instead are placed on the special //wait queues// in the kernel. These threads will become runnable again only when CPU usage drops below the cap level.
113: 
114:  The time spent by threads on //wait queues// is combined with the time spent on //run queues// and is reported as part of the //``wait-cpu’’// (latency) time by procfs. The //``wait-cpu’’// time is also used to report time spent waiting on //run queues// (see pr_wtime field of struct prusage in [[proc(4)>>http://docs.sun.com/app/docs/doc/816-5174/6mbb98uiq?a=view]] ). Wait times can be seen in the LAT column when [[prstat(1M)>>http://docs.sun.com/app/docs/doc/816-5166/6mbb1kqcm?a=view]] is invoked with the -m option. CPU time spent by threads on wait queues is also accumulated at the LMS_WAIT_CPU micro-state accounting state which is also used to account for time spent waiting on //run queues//.
115: 
116:  We decided to combine the wait times that threads spent waiting on //run queues// and //wait queues// into a single bucket because currently there is a fixed set of micro-state accounting types. Any extension of this set will cause offsets of fields in data structures, embedding micro-state data, to change. This, in turn, implies that all [[proc(4)>>http://docs.sun.com/app/docs/doc/816-5174/6mbb98uiq?a=view]] consumers should be recompiled. We feel that CPU wait time is general enough to include time spent waiting on both //run queues // and //wait queues// and it is reasonable to combine them together until micro-state accounting framework is extended.
117: 
118:  CPU caps are enforced only for threads running in //TS//, //IA//, //FX//, and //FSS// scheduling classes and have no effect on threads running in //RT// (Real-Time) scheduling class.
119: 
120:  All CPU usage accounting data is collected using micro-state accounting facility, so the accuracy of CPU caps depends on the accuracy of the micro-state data.
121: 
122:  A more detailed description of the implementation is available in the implementation guide [[[2>>#bib:caps:impl]]].
123: 
124: =   5 CPU Caps Observability
125: 
126:  There are several ways to observe the impact of CPU caps at a zone or project level and at a process or thread level. Each project or zone CPU cap exports information via kstats (see [[5.1>>#observability:kstat]]).
127: 
128:  The DTrace sched provider is extended with two new probes which can be used for gathering detailed data for times spent on wait queues (see [[5.3>>#dtrace]]). These probes provide thread and process level observability for CPU caps.
129: 
130: ==   5.1 Zones and Project observability
131: 
132:  Zone and project CPU cap kstats contain the following information:
133: 
134: ; **value**
135: : - the cap value in percentages of a single CPU
136: ; **usage**
137: : - current aggregated CPU usage for all threads belonging to a capped project or zone in percentages of a single CPU
138: ; **maxusage**
139: : - maximum observed CPU usage
140: ; **above_sec**
141: : - total time in seconds spent above the cap
142: ; **below_sec**
143: : - total time in seconds spent below the cap
144: ; **nwait**
145: : - number of threads on cap wait queue
146: 
147:  For example, when a project CPU cap is set to 50% for project 1234, the following command will show cap information:
148: 
149: {{{
150: 
151: $ kstat -m caps
152: module: caps instance: 0
153: name:   cpucaps_project_1234 class: project_caps
154:         above_sec    787
155:         below_sec    260551
156:         nwait        0
157:         usage        1
158:         maxusage     51
159:         value        50
160: 
161: }}}
162: 
163:  The following example is for zone kstats:
164: 
165: {{{
166: 
167: module: caps instance: 14
168: name:   cpucaps_zone_14 class: zone_caps
169:         above_sec  0
170:         below_sec  3
171:         maxusage   255
172:         nwait      0
173:         usage      19
174:         value      300
175: 
176: }}}
177: 
178:  For both zone and project kstats, the kstat instance is the same as zone ID. The PSARC 2006/598 //Swap resource control// case[[[4>>#PSARC2006598]]] provides a precedents for such kstats and establishes the naming conventions.
179: 
180:  The maximum usage statistics provides a good way for users to estimate how to set up their CPU caps. They can set the cap to a very high value and observe the maximum usage while their application is running for a while. This gives an estimate of maximum CPU requirements for the workload.
181: 
182: ==  5.2 Thread level observability
183: 
184:  The [[ps(1)>>http://docs.sun.com/app/docs/doc/816-5165/6mbb0m9pk?a=view]] command shows threads on the wait queue by displaying ``W’’ for their state. For example:
185: 
186: {{{
187: 
188: $ ps -o pid,s,comm -p 101262
189:    PID S COMMAND
190: 101262 W /usr/perl5/bin/perl
191: 
192: }}}
193: 
194:  The [[prstat(1M)>>http://docs.sun.com/app/docs/doc/816-5166/6mbb1kqcm?a=view]] command shows //``wait’’// state for threads, sitting on wait queues. For example:
195: 
196: {{{
197: 
198:    PID USERNAME SIZE  RSS  STATE  PRI NICE      TIME  CPU PROCESS/NLWP
199: 100686 akolb   4272K 1516K wait     0    0   0:51:20  25% perl/1
200: 
201: }}}
202: 
203: ==   5.3 DTrace changes for CPU caps
204: 
205:  CPU caps provide two new DTrace sched provider probes for observing scheduling impact for threads and processes:
206: 
207: ; **cpucaps-sleep**
208: : Probe that fires immediately before the current thread is placed on a wait queue. The lwpsinfo_t of the waiting thread is pointed to by args[0]. The psinfo_t of the process containing the waiting thread is pointed to by args[1].
209: ; **cpucaps-wakeup**
210: : Probe that fires immediately after a thread is removed from a wait queue. The lwpsinfo_t of the waiting thread is pointed to by args[0]. The psinfo_t of the process containing the waiting thread is pointed to by args[1].
211: 
212: =  6 Public Documentation Changes
213: 
214: ==   6.1 [[resource_controls(5)>>http://docs.sun.com/app/docs/doc/816-5175/6mbba7f37?a=view]]
215: 
216:  The following description should be added to [[resource_controls(5)>>http://docs.sun.com/app/docs/doc/816-5175/6mbba7f37?a=view]] man page:
217: 
218: ; **project.cpu-cap**
219: :
220:  Maximum amount of CPU resources that a project can use. The unit used is the percentage of a single CPU (an integer). The cap does not apply to threads running in real-time scheduling class.
221: ; **zone.cpu-cap**
222: :
223:  Sets a limit on amount of CPU time that can be used by a zone. The cap does not apply to threads running in real-time scheduling class. Projects within the zone can have their own CPU caps. The minimum cap value takes precedence. Expressed as an integer, denoting the percentage of a single CPU that can be used by all user threads in a zone.
224: 
225: ==  6.2 Solaris Dynamic Tracing Guide
226: 
227:  The Solaris Dynamic Tracing Guide should be updated to include information about new process states and two new sched provider probes.
228: 
229: ===  6.2.1 proc Provider
230: 
231:  The proc //Provider// section of the guide should be updated to reflect the addition of the new SWAIT processor state ([[table 25-5>>http://docs.sun.com/app/docs/doc/817-6223/6mlkidll3?a=view#tbl-sched-state]] ):
232: 
233: ; **SWAIT(W)**
234: : The thread is waiting on wait queue. The sched:::cpucaps-sleep probe will fire immediately before a thread state is transitioned to SWAIT.
235: 
236: ===  6.2.2 Probes
237: 
238:  The //Probes// section of the guide ([[table 26-1>>http://docs.sun.com/app/docs/doc/817-6223/6mlkidll8?a=view#tbl-sched]] ) should be updated with information about two new probes:
239: 
240: ; **cpucaps-sleep**
241: : Probe that fires immediately before the current thread is placed on a wait queue. The lwpsinfo_t of the waiting thread is pointed to by args[0]. The psinfo_t of the process containing the waiting thread is pointed to by args[1].
242: ; **cpucaps-wakeup**
243: : Probe that fires immediately after a thread is removed from a wait queue. The lwpsinfo_t of the waiting thread is pointed to by args[0]. The psinfo_t of the process containing the waiting thread is pointed to by args[1].
244: 
245: ===  6.2.3 Arguments
246: 
247:  The //Arguments// section of the guide ([[table 26-2>>http://docs.sun.com/app/docs/doc/817-6223/6mlkidll9?a=view#tbl-sched-args]] ) should be updated with information in following table, describing arguments for CPU Caps probe arguments:
248: 
249: |**Probe**|args[0]|args[1]
250: |cpucaps-sleep|lwpsinfo_t *|psinfo_t *
251: |cpucaps-wakeup|lwpsinfo_t *|psinfo_t *
252: 
253: ===  6.2.4 cpucaps-sleep and cpucaps-wakeup Examples
254: 
255:  The //Examples// section should be updated with examples for CPU Caps probe arguments.
256: 
257: =   A Summary of exported interfaces
258: 
259:  Here is a summary of proposed interface changes. All of them, except kstats are //Committed//.
260: 
261: * New resource controls:
262: ** project.cpu-cap
263: ** zone.cpu-cap
264: * [[zonecfg(1M)>>http://docs.sun.com/app/docs/doc/816-5166/6mbb1kqm2?a=view]] extensions:
265: ** capped-cpu resource
266: ** ncpus property
267: * zone and project kstats for CPU caps
268: * [[proc(4)>>http://docs.sun.com/app/docs/doc/816-5174/6mbb98uiq?a=view]] consumers:
269: ** ``W’’ state shown by [[ps(1)>>http://docs.sun.com/app/docs/doc/816-5165/6mbb0m9pk?a=view]] for threads on wait queues
270: ** ``sleep’’ state shown by [[prstat(1M)>>http://docs.sun.com/app/docs/doc/816-5166/6mbb1kqcm?a=view]] for threads on wait queues
271: * DTrace extensions:
272: ** cpucaps-sleep sched provider probe
273: ** cpucaps-wakeup sched provider probe
274: ** SWAIT processor state for threads on wait queues
275: 
276: ==  References
277: 
278: :
279: ; 1
280: :  CPU caps web page on OpenSolaris.org
281: [[http://www.opensolaris.org/os/project/rm/rctls/cpu-caps/>>Project rm.cpu-caps]]
282: ; 2
283: :  Implementation description
284: [[http://www.opensolaris.org/os/project/rm/rctls/cpu-caps/caps_implementation/>>Project rm.caps_implementation]].
285: ; 3
286: :  PSARC/2006/496 Improved Zones/RM Integration
287: [[http://sac.sfbay.sun.com/PSARC/2006/496>>http://sac.sfbay.sun.com/PSARC/2006/496]]
288: [[http://www.opensolaris.org/os/community/arc/caselog/2006/496>>Community Group arc.496]]
289: ; 4
290: : PSARC 2006/598 Swap resource control; locked memory RM improvements.
291: [[http://sac.sfbay.sun.com/PSARC/2006/598>>http://sac.sfbay.sun.com/PSARC/2006/598]]
292: [[http://www.opensolaris.org/os/community/arc/caselog/2006/598>>Community Group arc.598]]
293: ; 5
294: :  On-line version of CPU Caps PSARC case
295: [[http://www.opensolaris.org/os/project/rm/rctls/cpu-caps/caps_psarc/>>Project rm.caps_psarc]]
296: 
297: ----
298:  Alexander Kolbasov 2007-01-10
OpenSolaris

Top Menu

Wiki code for CPU Caps PSARC case draft

Search

Collectives

Community Group

Project

User Group

Subsites

Project rm Pages
