Memory Sets

Memory Sets

The memory sets feature is current under development within Sun.  It extends the pools framework, adding the "mset" resource set type.

Overview

Memory sets will enable administrators to partition physical memory into reservations for various entities such as projects and zones.  Eventually memory sets will also include partitioning system swap.  This topic illustrates the current interface design for the feature.

Example: Partitioning a system for 4 equal zones (1 global + 3 non-global).

In this example, assume a system with:

• 2 Gb of Physical Memory (RAM)
• 4 Gb of disk swap
• 4 cpus (4 chips, or 4 cores, or 4 hardware threads, etc).

The goal is to divide the system resources up evenly among 4 zones.  This example makes use of various resource management features to partition the system, including memory sets.

The mset functionality in this example is actively under development and not integrated into OpenSolaris.  This example serves to illustrate the current interface design.  The actual procedure in this example may differ when the feature integrates.

Procedure Overview.

The procedure is broken up into these steps:

1.  Configure and Install Zones.
2.  Create a resource pool for each zone.
3.  Create a processor set and memory set for each resource pool.
4.  Configure additional resource controls on each zone.
5.  Boot the Zones.

Step 1, Configure and Install Zones

For the purpose if this example, assume four zones have been configured and installed.  The zones are named "zone1" - "zone3", and the "global" zone is
used.  Here is a good simple example by Brendan Gregg for setting up a zone.

Step 2, Create a resource pool for each Zone.

Each zone will have its own resource pool.  The global zone will use pool_default, which always exists when pools are enabled.  For the other zones, "pool1" - "pool3" will be created.

• Create the three pools:

    # svcadm enable pools
    # pooladm -s
    # for N in 1 2 3 ; do 
    >     poolcfg -c "create pool pool$N"
    > done
    # pooladm -c

• Configure each zone to bind to its pool upon boot.  The global zone is already by default bound to pool_default.

    # for N in 1 2 3 ; do
    >     zonecfg -z zone$N set pool=pool$N
    > done

Step 3, Create a processor and memory set for each resource pool.

Create a processor set and memory set for each pool.  Each pool will receive
1 cpu, 512 Mb of RAM, and 1Gb of swap:

    # for N in 1 2 3 ; do
    > poolcfg -f - <<EOF
    > create pset pset$N (uint pset.min = 1      ; uint pset.max = 1)
    > create mset mset$N (uint mset.min = 512M   ; uint mset.max = 512M)
    > modify mset mset$N (uint mset.minswap = 1GB; uint mset.maxswap = 1GB)
    > associate pool pool$N (pset pset$N ; mset mset$N)
    > EOF
    # pooladm -c

Step 4, Configure additional resource controls for each zone.

• zone.max-lwps
• zone.max-processes (not yet implemented in OpenSolaris)

These resource controls will limit the number of threads and processes that can be created by each zone.  Since threads and processes consume kernel memory, it is important to limit them to protect against fork/thread/zombie bombs.  Depending on the intended workloads for these zones, set these accordingly.
For examples on setting zone resource controls, see Zones and Resource Management.

Step 5, Boot the Zones.

    # for N in 1 2 3 ; do
    > zoneadm -z zone$N boot
    > done

Pool bindings and resource controls are set when a zone boots.  poolbind(1m)
and prctl(1) can be used to set these on an already booted zone.  See Zones and Resource Management.

Resource controls not used in this example.

• zone.cpu-shares
• zone.cpu-cap (not yet available in OpenSolaris)

These resource controls are useful when multiple projects or zones are sharing
the same processor set, and scheduling fairness or absolute limits on cpu cycle
consumption is desired.  In this example, each zone has its own dedicated pset,
to these controls are not needed.

• zone.max-locked-memory (not yet available in OpenSolaris) 

This resource control is not used because a memory set implicitly limits locked memory.  If multiple zones are configured to share the same memory set, then zone.max-locked-memory will prevent a zone from locking all the memory in the set.

Future Work

Potential areas in which solaris could be enhanced to partition system resources:

• Swap device isolation for memory sets and/or zones.
• Partitioning of io bandwidth
• Partitioning of system (bus) bandwidth.
• Limits on kernel memory allocated on the behalf of an entity such as a zone or project.