OpenSolaris

lgrpinfo(1): tool for displaying lgroup hierarchy

The lgrpinfo command prints information about the locality group (lgroup)
hierarchy, contents, and characteristics.

An lgroup represents the set of CPU and memory hardware devices that are at most
some distance (latency) apart from each other. All lgroups in the system are
identified by a unique integer called an lgroup ID. The lgroups are organized
into a hierarchy to facilitate finding the nearest resources. Leaf lgroups
contains set of resources that are closest to each other. Each parent lgroup in
the hierarchy contains the resources of its child lgroups plus their next
nearest resources. Finally, the root lgroup contains all the resources in the
domain within the largest latency.

A Uniform Memory Access (UMA) machine will simply be represented by the root
lgroup. A Non Uniform Memory Access (NUMA) machine is represented by a hierarchy
of lgroups to show the corresponding levels of locality. For example, a NUMA
machine with two latencies (local and remote) will have an lgroup hierarchy
consisting of two levels with its leaves and the root.

Every application thread is assigned a home lgroup. When the system needs to
allocate a CPU or memory resource for the thread, it searches for available
resources in its home lgroup and continues up the lgroup hierarchy until it
finds one if necessary.

The lgrpinfo(1) is a Perl script that requires Solaris::Lgrp module. It uses  liblgrp(3LIB) API to get the information from the system and displays it in the human-readable form.

Example output of lgrpinfo -Ta on a 4-CPU AMD system:

    $ lgrpinfo -Ta
    0
    |~-- 5
    |   CPUs: 0-2
    |   Lgroup resources:   1-3 (CPU);  1-3 (memory)
    |   latency: 88
    |   {{code}}~-- 1
    |       CPU: 0
    |       Memory: installed 3839 Mb,  allocated 474 Mb,  free 3365 Mb
    |       Load:    0
    |       latency: 54
    |~-- 6
    |   CPUs: 0 1 3
    |   Lgroup resources:   1 2 4 (CPU);  1 2 4 (memory)
    |   latency: 88
    |   {{/code}}~-- 2
    |       CPU: 1
    |       Memory: installed 4096 Mb,  allocated 26 Mb,  free 4070 Mb
    |       Load:    0
    |       latency: 54
    |~-- 7
    |   CPUs: 0 2 3
    |   Lgroup resources:   1 3 4 (CPU);  1 3 4 (memory)
    |   latency: 88
    |   {{code}}~-- 3
    |       CPU: 2
    |       Memory: installed 4096 Mb,  allocated 14 Mb,  free 4082 Mb
    |       Load:    1
    |       latency: 54
    {{/code}}~-- 8
        CPUs: 1-3
        Lgroup resources:   2-4 (CPU);  2-4 (memory)
        latency: 88
        `~-- 4
            CPU: 3
            Memory: installed 4096 Mb,  allocated 16 Mb,  free 4080 Mb
            Load:    0
            latency: 54

Documents

• lgrpinfo(1) man page

• PSARC proposal (draft)

NOTE: The lgrpinfo tool is provided under the rems of the OpenSolaris CDDL License.

Code

• Perl script

Issues

• Need to verify that it works correctly on systems which have nodes without CPUs or memory

- lgrpinfo -m -u <unit> rounds to nearest integer <unit>, so it will print 0 Gb when there is 512 Mb if <unit> is "g" (for gigabytes).  Should it display fractional sizes (eg. 0.5 Gb instead of 0 Gb for 512 Mb)? (Will not fix)

• Confusing when latency info is printed or not printed.  Lgroup latency values printed by default and latency table printed with -l except with -T (Fixed with the introduction of -L option)

• Need to fix usage message and man page to explain default set of options correctly (Fixed)

• Usage message has spelling errors (eg. "lgoups" instead of "lgroups") (Fixed)

• Need to document -L option and behavior of -C, -P, and -I with no arguments
    (Fixed)

• Default behavior doesn't match man page (Fixed)
• Does -t -c -m -u m -r -l all with no options (this should include -e too)
• Does -L with -Ta

• Need to update examples in man page (Fixed)

• Prints memory and latency for root lgroup on UMA machine.  Shouldn't it print CPUs and should it print what it does for a leaf since root is the root and a leaf lgroup at the same time?  Need to just print 0 to be consistent with -T on NUMA machines.

• Display caller's view by default and OS view of lgroups when -G option given (Fixed)

Comments

Please send your comments and suggestions to the discussion forum.