OpenSolaris
en

OpenSolaris

Versions

  • v0.x.  There were many versions never made public that supported Solaris 10 and OpenSolaris.
  • v1.0.  Initial public preview version supporting OpenSolaris 2009.06.
  • v1.1.  Fixed a number of bugs and unified the setup commands as iscadm.ksh.
  • v1.2.  Unified hardening code for global zone and non-global zone uses, integrated with OpenSolaris JeOS project.
  • v1.3.  Added Crossbow anti-spoofing protections (when available), user-defined node network configurations, quotas/reservations and LWP resource controls for nodes.  Lots of cleanup and bug fixes.  Tested on b127.
  • v1.3.1.  Added default use of CPU shares (resource control) using the Fair Share Scheduler (FSS).

Functionality

The current build of the Immutable Service Container construction kit creates an environment satisfying the following requirements:

  • built upon the OpenSolaris 2009.06 release (also tested against b127)
  • (optional) loose minimization support to help those building and sharing images (OpenSolaris 2009.06 only)
  • security hardening of the operating system
  • non-executable stack functionality enabled (on systems supporting this functionality)
  • encrypted swap enabled
  • encrypted scratch space enabled
    • default size is 100 Mbytes (customize as needed)
  • kernel-level auditing enabled
    • default policy audits login/logouts, administrative events, and all commands executed on the system
    • audit syslog plugin configured (/var/log/auditlog)
  • configurable virtual network architecture
    • non-global zones can be installed into the same or different virtual networks
  • stateful packet filtering enabled
    • packet filtering syslog plugin configured (/var/log/ipflog)
    • in-bound network access denied by default (except SSH)
    • out-bound network access permitted by default (customize as needed)
  • a single non-global zone installed
    • gzip compressed non-global zone root file system
    • quota (default: 1G) and reservations (default: 512M) enabled by default on non-global zone root file system
    • per-non-global zone maximum lightweight process resource control installed (default: 300 LWPs each)
    • global zone and per-non-global zone CPU shares resource control installed (default: 100 shares each)
    • building upon default non-global zone security capabilities
    • unique VNIC limiting visibility of unintended network traffic
    • encrypted scratch space
    • stateful packet filtering and NAT to restrict network-access
    • anti-spoofing protection - network access denied if MAC or IP address is changed
    • in-bound network access denied by default (customize for your service)
    • out-bound network access permitted by default (customize as needed)
    • DNS and auditing configurations inherited from the global zone

When built, an ISC configuration is intended to be used as a virtual single system.  The global zone (Dock) performs administrative and monitoring functions similar to those of a system controller whereas all end-user services and functions should be installed into the non-global zone (Node).  In this way, services such as packet filtering, NAT and auditing can operate without being exposed to services or users operating inside of the non-global zone.  This enables greater operational integrity as those services and users are not able to alter the configurations or logs associated with these services.  Additional non-global zones can be added as needed.  This configuration uses a single exposed network interface and IP address for all of its communication even though internally services are separated to run inside of their own non-global zones.

Once built, organizations can further customize the configuration based upon their requirements to add things such as additional resource controls, read-only and read-write file systems (to the non-global zone), specific users and services, etc.

Functional Diagram

isc-arch-opensolaris-v1.3.png

Additional network architecture models can be found on this project's networking page.

Warning

This is pre-alpha (preview) software that is still undergoing substantial testing.  The goal in making it available is to increase the number of potential reviewers so that further feedback can be collected.  It is recommended that this software not be used on production systems at this time.  Further, this software does not have an undo or uninstall capability, so only use this software on a system that can be devoted to ISC testing - such as a virtual system.

The following OpenSolaris bugs have been encountered during testing of the ISC Construction Kit on OpenSolaris:

  • 6914378 AUT_SUBJECT64_EX token is incorrectly parsed by audit_syslog(5) and auditreduce.  This bug generates messages in syslog similar to error before token 6 (previous token=124) of record type execve(2).  Until this bug is fixed, some audit records may be incorrectly parsed by audit_syslog and auditreduce.
  • 6776009 nwam doesn't bring up pseudo data-links or flows at boot time.  To work around this bug, NWAM has been disabled by the OpenSolaris ISC Toolkit which then manually configured DHCP on the primary network interface.  This code will be removed when this bug is fixed and the default NWAM configuration will be maintained.  Note that this change was integrated into OpenSolaris build 132.
  • 6725151 zpool destroy core dumped with FAULTED zpool.  This bug is not triggered consistently.  It may appear as a core dump at system boot related to the encrypted scratch SMF service.  Rebooting the system often clears this issue.  It is expected that once this OpenSolaris bug is fixed, this will not be an issue.
  • 4838381 {task|zone}.max-lwps should include zombie threads in count.  This bug is triggered by certain denial of service attack methods (that create large numbers of zombie threads).  The result is that the number of LWPs used by a zone may exceed its limit as defined by zone.max-lwps.  Once this OpenSolaris bug is fixed, this will not be an issue.

Dependencies

This software currently depends upon the OpenSolaris 2009.06 release.  Features from this release used by this project include Crossbow Virtual Networking and encrypted LOFI.  This software has been tested on OpenSolaris build 127 and on systems where the functionality is available, Crossbow anti-spoofing functionality will be installed by default.

Minimization

Organizations wishing to deploy reduced or minimized configurations are encouraged to leverage the work of the Just Enough OS (JeOS) project.  This project is focused on delivering examples of heavily reduced, text-based, headless server-oriented forms of the Sun-managed OpenSolaris distribution such that developers and administrators can more easily realize application-tailored installations of OpenSolaris in virtual appliance, cloud and bare metal environments.

Using tools available from the JeOS project, one can either build a customized reduced installation profile or more simply use one of the existing JeOS images made freely available.  By using a JeOS installation as the foundation for the construction of the ISC, the OS footprint was able to be significantly reduced (from over 3 GB to just over 650 MB) without any loss of ISC functionality.  Additional software packages that are required can be installed as needed.

Note that some of the changes enabled by the OpenSolaris ISC Construction Kit may already be enabled in some JeOS images.  In those cases, system hardening may have been implemented in addition to enabling encrypted swap and auditing.  In these situations, organizations may choose to only implement the ISC steps that were not already implemented.  Check the documentation associated with the image that you are using to determine if this is the case.

For the initial testing, the OpenSolaris 2009.06 VirtualBox JeOS Image was used in conjunction with the steps outlined below to create a minimized ISC configuration.

Download

Currently, this software is accessible from a Mercurial source code repository.  To download a copy of the source, you can use a command similar to:

      opensolaris$ hg clone ssh://anon@hg.opensolaris.org//hg/isc/src isc
      requesting all changes
      adding changesets
      adding manifests
      adding file changes
      added 33 changesets with 83 changes to 37 files
      updating working directory
      29 files updated, 0 files merged, 0 files removed, 0 files unresolved

The OpenSolaris ISC Construction Kit distribution will be installed into the directory isc.  Note that the Mercurial tool set is not installed by default with OpenSolaris.  To install the Mercurial tools, use the following command:

     opensolaris$ pfexec pkg install SUNWmercurial

Pre-Installation

Since Immutable Service Container deployments are typically used in headless server environments, the graphical windowing system is disabled as part of the hardening functionality implemented by the OpenSolaris ISC Construction Kit.  If a graphical windowing system is used, then it is strongly recommended that the following command be executed from a remote terminal session or from the non-graphical system console.  To disable the windowing system, use the following command:

     opensolaris$ pfexec svcadm disable gdm

Installation

The OpenSolaris ISC Construction Kit can be run using the following command:

      opensolaris$ pfexec isc/bin/iscadm.ksh

Note that starting in version 1.3 of the OpenSolaris ISC Construction Kit, the default network number used by ISC nodes is 192.168.1.0.  In some cases, this network number may conflict with existing network addressing being used within an organization.  The actual network number can be changed using the -N option.  For example, to create an ISC node on network 192.168.0.0, use the following command:

      opensolaris$ pfexec isc/bin/iscadm.ksh -N 0

Note that the Network Auto-Magic project is not compatible with Crossbow Virtual Networking (as noted in Bug ID 6776009 above), so Network Auto-Magic is disabled and DHCP is configured for the primary network interface.  If this is not desired, the network configuration can be changed manually using standard OpenSolaris commands after the iscadm.ksh script has completed.

Once this command has completed, update the boot archive and shutdown the system:

      opensolaris$ pfexec bootadm update-archive
      opensolaris$ pfexec shutdown -g 0 -i 0 -y

When the system is restarted, it should now be in a fully operational ISC configuration.  Note that the non-global zone is installed with a default root password - iscroot.  It should be changed post-installation.  That said, there is no way to directly contact the non-global zone as it is not exposing any network services by default.

Also, note that upon first boot, the freshly installed zone will go through its normal initial system configuration process.  As a result, it may be a few minutes until the zone is fully ready to use.  You can monitor its progress by connecting to the zone console using the command:

     opensolaris$ zlogin -C isc1

It should be noted that additional non-global zones can be added using the following command:

      opensolaris$ pfexec isc/bin/iscadm.ksh -n # [-N #]

where -n is used to specify the number assigned to the non-global zone (e.g., 2, 3, 4, etc.) and (as noted above) -N is used to specify the number of the virtual network to which the non-global zone is assigned.  The first non-global zone created (by default) is number 1.

Selective Installation

With the OpenSolaris ISC Construction Kit, it is possible to only configure or enable specific elements while not enabling the full suite of functionality.  This is achieved using the ISC_SVCS_DOCK and ISC_SVCS_NODE parameters respectively.  The ISC_SVCS_DOCK variable determines which security configuration changes will be made to the ISC Dock (i.e, OpenSolaris Global Zone).  Similarly, the ISC_SVCS_NODE variable determines which security configuration changes will be made to the ISC Node (i.e., OpenSolaris Non-Global Zone).  To use these parameters, simply create the ${ISC_HOME}/etc/site.conf file, assigning the variables as necessary.  

For example, to enable just system hardening, auditing and encrypted swap (such as the configuration used in some JeOS images and Security Enhanced OpenSolaris AMIs), edit the ${ISC_HOME}/etc/site.conf file and add the following lines:

     ISC_SVCS_DOCK="encrypted_swap auditing lockdown"
     ISC_SVCS_NODE=""

Since an ISC Node is not being created, use the following command to apply these settings only to the ISC Dock:

     opensolaris$ pfexec isc/bin/iscadm.ksh -d

Pre-Configured Images

Open Virtualization Format Images

Using the steps above, a pre-installed and pre-configured OpenSolaris 2009.06 Immutable Service Container image has been published using the Open Virtualization Format (OVF).  The images are roughly 1.5 Gbytes and are made freely available (under a CDDL license) to those interested in evaluating the technology without having to go through the steps noted above.

To use the images, simply download and verify its file fingerprint:

     opensolaris$ digest -a md5 isc-demo-v1.0.tar.bz2
     15ca60c0435a012aa15dc04b850fee58

     opensolaris$ digest -a sha256 ./isc-demo-v1.0.tar.bz2
     77fa29b90a60a6e6bcb84e678a7323ce33adeec77dda3563c9460ea6d03a82f9

Once verified, import into VirtualBox or other OVF capable tool and start the virtual machine.  To access the system, use the iscadm account (with password iscadm).  The root password (available also from the iscadm account) is iscroot.

Note that these images have been modified using the steps below to add the Apache web server into the (only) non-global zone running on the system (isc1).  No other changes have been made outside those implemented by the OpenSolaris ISC Construction Kit.  As these images capture a moment in time, the OpenSolaris ISC Construction Kit software installed in these images may be out of date.  To update the software, use the following command as the iscadm account:

     opensolaris$ cd $HOME/isc; hg pull -u ssh://anon@hg.opensolaris.org//hg/isc/src 

Amazon Machine Image (AMI) Format

Using steps similar to those noted above, a pre-installed and pre-configured OpenSolaris 2009.06 Immutable Service Container image has been published for the Amazon Web Services Elastic Compute Cloud (EC2) environment.  There images are freely available in both the U.S. (AMI ID: ami-48c32021) and European (AMI ID: ami-78567d0c) regions.  For more information, there is also an article discussing how the OpenSolaris ISC Construction Kit should be configured when using it to create Amazon EC2 instances.

Adding Services and Applications

Service Installation and Configuration

Once a system's configuration has been modified using the OpenSolaris ISC Construction Kit, it is often useful to add services to the non-global zone in order to more fully test the configuration.  The following steps can be used to install the Apache server.  First, the actual web server software should be installed and enabled in the non-global zone:

      opensolaris$ pfexec zlogin isc1 pkg install SUNWapch22
      opensolaris$ pfexec zlogin isc1 svcadm enable apache22

Once the service has been installed and is running properly, the network filtering and NAT policy must be changed to allow the Apache server to be accessed.  The following commands must be run in the global zone:

  • Add a new IP Filter rule to allow access to 192.168.1.1:80 (TCP).  To do this, you will need to edit the /etc/ipf/ipf.conf file, copy the Node-specific entry for SSH (that is commented out), uncomment the new line, and change it to use port 80 instead of port 22.  It should look something like this:

      pass in quick on e1000g0 proto tcp from any to 192.168.1.1 port = 80 keep state

  • Next, add a IP NAT rule to the /etc/ipf/ipnat.conf file redirecting traffic from the external IP address to the non-global zone for port 80.  It should look something like this:

      rdr e1000g0 0.0.0.0/0 port 80 -> 192.168.1.1 port 80

  • Lastly, enable this new policy by executing the following commands:

      opensolaris$ pfexec ipf -Fa -f /etc/ipf/ipf.conf
      opensolaris$ pfexec ipnat -FC -f /etc/ipf/ipnat.conf

With this new policy in effect, you should now be able to reach the web server (running on port 80 in the non-global zone) using the IP address of the global zone (obtained via DHCP).  Note that since we can leverage port-level redirection, we no longer need to use reserved network ports for services such as HTTP since we can simply redirect incoming traffic.  We could start our web server on port 8080 or some other non-privileged port while still exposing the traditional port (e.g., TCP/80) to the outside world.  Why configure the service in this manner?  Simple - it allows us to start a service without ever having to give it administrative privileges (such as net_privaddr).  It should be noted that services should be deployed using least privilege where possible and should themselves be security hardened for additional protection.

Service Security Tuning

In addition to simply installing a new service into a non-global zone, you may also want to perform the following operations:

  • Use unique credentials and least (process) privileges
  • Optimize the security configuration of the new service
  • Leverage encrypted scratch space for sensitive content
  • Consider using a custom file system layout for immutability
    • Read only: binaries, libraries, etc.
    • Read write: logs, configuration files and data (if needed)

Additional steps can be taken from the global zone including:

  • Implement additional resource controls for the service non-global zone

Alternative Networking Configurations

In addition to the process of adding additional ISC Nodes to one or more Private Virtual Networks, it is possible to create a variety of network configurations depending upon your requirements.  Several representative network architecture diagrams are available.  The following diagram illustrates the OpenSolaris implementation (networking view) of the Multiple Instance, Mediated Private Virtual Network with Unique Public IPs network model.  

isc-osol-net-multi-mediated-unique-v1.3.png

No changes to the OpenSolaris ISC Construction Kit are required to create this model although there are some additional post-installation steps that must be completed in order to (1) adjust the IP Filter and IP NAT configuration and (2) create the new public network interfaces and assign addresses to them.  These steps are implemented using the usual OpenSolaris administrative practices:

     opensolaris$ pfexec cp /etc/ipf/ipnat.conf /etc/ipf/ipnat.conf.ORIG
     opensolaris$ pfexec /bin/sh -c 'cat > /etc/ipf/ipnat.conf' << EOL_IPNAT
     map e1000g0 192.168.1.1 -> 10.10.1.1/32 portmap tcp/udp auto
     map e1000g0 192.168.1.1 -> 10.10.1.1/32
     rdr e1000g0 10.10.1.1/32 port 80 -> 192.168.1.1 port 80
    
     map e1000g0 192.168.2.2 -> 10.10.1.2/32 portmap tcp/udp auto
     map e1000g0 192.168.2.2 -> 10.10.1.2/32
     rdr e1000g0 10.10.1.2/32 port 80 -> 192.168.2.2 port 80
    
     map e1000g0 192.168.3.3 -> 10.10.1.3/32 portmap tcp/udp auto
     map e1000g0 192.168.3.3 -> 10.10.1.3/32
     rdr e1000g0 10.10.1.3/32 port 80 -> 192.168.3.3 port 80
     EOL_IPNAT
    
     opensolaris$ pfexec cp /etc/ipf/ipf.conf /etc/ipf/ipf.conf.ORIG
     opensolaris$ pfexec /bin/sh -c 'cat > /etc/ipf/ipf.conf' << EOL_IPF
     # Allow SSH and DHCP inbound and anything outbound.
     pass in quick on e1000g0 proto udp from any port = 67 to any port = 68 keep state
     pass out quick on e1000g0 from e1000g0/32 to any keep state
     pass in quick on e1000g0 proto tcp from any to e1000g0/32 port = 22 keep state
    
     # Ensure that ISC Nodes can't talk to one another.
     block in log quick on isc_0_254 from 192.168.1.0/24 to 192.168.2.0/24 keep state
     block in log quick on isc_0_254 from 192.168.1.0/24 to 192.168.3.0/24 keep state
     block in log quick on isc_2_254 from 192.168.2.0/24 to 192.168.0.0/24 keep state
     block in log quick on isc_2_254 from 192.168.2.0/24 to 192.168.3.0/24 keep state
     block in log quick on isc_3_254 from 192.168.3.0/24 to 192.168.0.0/24 keep state
     block in log quick on isc_3_254 from 192.168.3.0/24 to 192.168.2.0/24 keep state
    
     # Configure ISC Nodes to accept connections on TCP/80 inbound, allow all outbound.
     pass in log quick on isc_1_254 from 192.168.1.1 to any keep state
     pass in log quick on e1000g0 from any to 192.168.1.1 port = 80 keep state
     pass out log quick on e1000g0 from 192.168.1.1 to any keep state
     pass in log quick on isc_2_254 from 192.168.2.2 to any keep state
     pass in log quick on e1000g0 from any to 192.168.2.2 port = 80 keep state
     pass out log quick on e1000g0 from 192.168.2.2 to any keep state
     pass in log quick on isc_3_254 from 192.168.3.3 to any keep state
     pass in log quick on e1000g0 from any to 192.168.3.3 port = 80 keep state
     pass out log quick on e1000g0 from 192.168.3.3 to any keep state
    
     # Drop everything else.
     block in log all
     block in from any to 255.255.255.255/32
     block in from any to 127.0.0.1/32
     EOL_IPF
    
     opensolaris$ pfexec chown root:sys /etc/ipf/ipf.conf /etc/ipf/ipnat.conf
     opensolaris$ pfexec chmod 644 /etc/ipf/ipf.conf /etc/ipf/ipnat.conf
    
     opensolaris$ pfexec /bin/sh -c 'echo 10.10.1.1 > /etc/hostname.e1000g0:1
     opensolaris$ pfexec /bin/sh -c 'echo 10.10.1.2 > /etc/hostname.e1000g0:2
     opensolaris$ pfexec /bin/sh -c 'echo 10.10.1.3 > /etc/hostname.e1000g0:3
    
     opensolaris$ pfexec chown root:sys /etc/hostname.e1000g0:*
     opensolaris$ pfexec chmod 644 /etc/hostname.e1000g0:*

Upgrading

Initial tests have shown that it is possible to upgrade an Immutable Service Container using the standard OpenSolaris packaging mechanisms.  During these tests, the OpenSolaris 2009.06 operating system was upgraded from OpenSolaris 2009.06 to OpenSolaris build 130 (from the development tree).  Post upgrade verification showed that all of the core security configuration changes remained intact.  Note that these tests should still be considered preliminary and care should be taken if you choose to upgrade (vs. re-create) your ISC configurations.  The process that was tested included the following steps:

     opensolaris$ pfexec pkg set-authority -P -O http://pkg.opensolaris.org/dev/ opensolaris.org
     opensolaris$ pfexec pkg install SUNWipkg
     opensolaris$ pfexec pkg image-update

In addition to the global zone, each of the individual non-global zones must also be updated:

     opensolaris$ pfexec zoneadm -z isc1 halt
     opensolaris$ pfexec mount -F zfs rpool/export/isc/isc1/zone/ROOT/zbe-1 /mnt
     opensolaris$ pfexec pkg -R /mnt set-authority -P -O http://pkg.opensolaris.org/dev/ opensolaris.org
     opensolaris$ pfexec pkg -R /mnt install SUNWipkg
     opensolaris$ pfexec pkg -R /mnt image-update
     opensolaris$ pfexec umount /mnt

Once completed, update the boot archive and reboot the system:

     opensolaris$ pfexec bootadm update-archive
     opensolaris$ pfexec reboot

Other Directions

Related Projects

Selected References

Additional Reading

Web Pages, Blogs

White Papers

Presentations

Immutable Service Containers

Supporting Areas

Other Media

Software and Tools

Additional Resources

Created by Glenn Brunette on 2010/02/03 14:11
Last modified by Glenn Brunette on 2010/02/12 18:14

Collectives

Project isc Pages

XWiki Enterprise 2.7.1.34853 - Documentation