iSER TOI

iSER TOI

 This page is a modified version of a Sun-internal "Transfer of Information" document. It's intended to serve as a complement to the other iSER material on this site.

1. Overview

List overall project requirements and the requirements for various components involved in this project

 iSER accelerates the iSCSI protocol by mapping the data transfer phases to Remote DMA (RDMA) operations. As a result an iSER initiator should be able to read and write data from an iSER target at high data rates with relatively low CPU utilization compared to iSCSI using TCP/IP. This project will implement both an iSER initiator and an iSER target. In addition, the project will implement an iSCSI port provider (iscsit) for COMSTAR adding an iSCSI transport option to the existing Fibre Channel and SAS support. The iSER target will be an extension of this iSCSI port provider.

List the requirements/features that are not currently supported

 RDMA transports other than infiniband are not currently supported.

Give a brief overview of the components and what is being added, changed or deleted as part of this project

• iSCSI Initiator modified to use IDM
• iSCSI COMSTAR port provider (iscsit)
• iSCSI Data Mover kernel module (IDM)
• iSER transport plugin for IDM

List all the dependencies this project has on other projects

 This project depends on the existing COMSTAR and Infiniband driver projects.

Provide any constraints or limitations associated with this project

• Single connection per session.
• Error recovery level 0.
• No support for iSCSI or socket tunables.

List the components developed and how long they need to be supported in the field

 The current iSER design defines a component called "iSCSI Data Mover" (IDM). The iSCSI initiator and iSCSI target port provider will link against the IDM kernel module which will provide iSCSI connection services and a transport abstraction layer. iSER will be implemented beneath this abstraction layer allowing both initiator and target to use a single shared iSER implementation.

 The project will deliver the following components:

• iSCSI Initiator modified to use IDM
• iSCSI COMSTAR port provider (iscsit)
• iSCSI Data Mover kernel module (IDM)
• iSER transport plugin for IDM

1.1 High Level Design

Provide a block diagram of all the components and explain how they interact

• sd: SCSI disk driver
• scsa: Sun Common SCSI Architecture
• iSER: iSCSI Extensions for Remote DMA
• IDM: iSCSI Data Mover
• IPoIB: IP over Infiniband
• IB HCA: Infiniband Host Channel Adaptor
• STMF: SCSI Target Mode Framework

List the interfaces exported by each components and how/when they are called from one component into another

• itadm CLI for iSCSI target management
• libiscsit

List the ARC cases associated with the project

 ARC case:

• PSARC/2008/395: ''iSER: RDMA Extensions for RDMA''

 Other relevant ARC cases:

• PSARC 2007/523 COMSTAR: Common Multiprotocol SCSI Target
• PSARC 2007/153 iSCSI target provider
• PSARC 2006/622 iSCSI/ZFS integration
• PSARC 2005/441 iSCSI Target Project

1.2 Low Level Design

For each component, explain in detail the functions of each interface/entry point

 This is explained in the following documents:

•  COMSTAR iSCSI Port Provider Design Document
•  iSCSI Data Mover Design document 
•  iSER Transport Design document

Explain in detail the protocols and algorithms involved in each component

iSER protocol

•  iSER RFC 5046: Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA) 

iSCSI target

•  iSCSI target state machine (last page is new target state machine)
•  iSCSI initiator cmd state machine  (state diagram is near the head of the file)

Explain in detail key data structures and algorithms.

Overview

 The best guide to the key data structures, locking, and synchronization is the code itself. What follows are some notes regarding particular aspects of the code.

idm hold and rele routines

 There are a number of idm_*_hold() and idm_*_rele() routines (e.g. idm_conn_hold(), idm_conn_rele()) defined in idm. If you call an idm lookup or find function it takes a reference. So every time you call idm_task_find() it is the same as calling idm_task_hold(). You must explicitly call idm_task_rele() after you are done working with the task.

 In general it is good to release the reference in the same function you acquired the hold. So if you call idm_task_find(), you should call idm_task_rele() before you leave the function. Similarly if you call idm_task_hold(), call idm_task_rele() before you leave the function. There are exceptions to this rule but consider these exceptions carefully. Following this rule helps to make the refcnts easy to review and validate.

List the websites/Wikis maintained for each component

 OpenSolaris iSER Project Page 

1.3 Source Gate

Provide the location of the master gate and all the clones

ssh://anon@hg.opensolaris.org/hg/iser/iser-onnv

Provide the location of code snapshots for binaries for various releases/builds

 These are listed on the  OpenSolaris iSER project page.

Describe the source gate structure

• usr/src/lib/libiscsit -- iscsi target library source
• usr/src/cmd/itadm -- itadm command
• usr/src/cmd/iscsitsvc -- iscsi target service
• usr/src/cmd/mdb -- walkers for iscsi
• usr/src/uts/common/io/comstar/port/iscsit -- COMSTAR iscsi target
• usr/src/uts/common/io/idm -- IDM
• usr/src/uts/common/io/ib/clients/iser -- IB iSER client support
• usr/src/uts/common/io/scsi/adapters/iscsi -- iscsi initiator

Describe how to build the entire gate (including packages) and do incremental builds

To perform a nightly build (required for putback to the project gate)

{{{
cd my-workspace
cp usr/src/tools/env/developer.sh .
}}}

 

 (Edit developer.sh for your parameters. If you don't know what to edit, ask an experienced OpenSolaris developer.)

{{{
nohup env nightly -n ./developer.sh &
}}}

 

To build individual components

{{{
cd my-workspace
ws
cd usr/src
make setup (This setups up header files and other prerequisites)
cd usr/src/lib/libiscsit; make install
cd /usr/src/cmd/itadm; make install
cd /usr/src/cmd/iscsitsvc; make install
cd usr/src/cmd/mdb; make install (Not sure if this will work... if not do a nightly)
cd usr/src/uts/intel/idm; make install (replace intel with sparc on sparc)
cd usr/src/uts/intel/iscsit; make install (replace intel with sparc on sparc)
cd usr/src/uts/intel/iser; make install (replace intel with sparc on sparc)
}}}

 

1.4 Installation

Provide pointers to installation guides

 The installation guide may be found in the  iSER Release Notes.

 Note that you may need to run add_drv iser after installing.

1.5 Configuration

Provide pointer to configuration guides

 A configuration guide may be found in the  iSER Release Notes.

1.6 Debugging

Provide details on how to debug each component

 Debugging tips may be found on the iSER Debugging Tips page.

 A list of relevant MDB dcmds may be found on the MDB dcmds page.

1.7 Release Mechanism

Provide details on how the software developed for different components

 The software is developed as a number of Solaris packages.

List all the packages and what goes into each package

• SUNWib: Sun InfiniBand Framework
• SUNWibsdp: Sun InfiniBand layered Sockets Direct Protocol
• SUNWibsdpib: Sun InfiniBand Sockets Direct Protocol
• SUNWibsdpu: Sun InfiniBand Sockets Direct Protocol Admin
• SUNWipoib: Sun IP over InfiniBand
• SUNWiscsidmr: Sun iSCSI Data Mover (root)
• SUNWiscsidmu: Sun iSCSI Data Mover (usr)
• SUNWiscsir: Sun iSCSI Device Driver (root)
• SUNWiscsitr: Sun iSCSI COMSTAR Port Provider (root)
• SUNWiscsitu: Sun iSCSI COMSTAR Port Provider (usr)
• SUNWiscsiu: Sun iSCSI Management Utilities (usr)
• SUNWistmf: Sun Common Multiprotocol SCSI Target
• SUNWrpcib: InfiniBand plugin implementation of RPC over RDMA
• SUNWstmfu: Sun Common Multiprotocol SCSI Target Libraries and Tools

Provide details on how to get release notes for various releases

 Release notes for development versions may be found here.

List the dependencies between different releases of various components and their support matrix

 All components are intended to be from the same release of OpenSolaris.

1.8 Bug Tracking

List Bugster product/cat/subcat used for different components including test suites

• product = Solaris
• category = storage_sw
• subcategories = idm, iscsit, iser, itadm, libiscsit

1.9 Documentation

List the man pages available

• iscsiadm(1M)
• sbdadm(1M)
• stmfadm (1M)

List the industry standard specifications and documents/specifications from external vendors

•  RFC 5046: Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA)
•  RFC 3720: Internet Small Computer Systems Interface (iSCSI)
•  RFC 5047: DA: Datamover Architecture for the Internal Small Computer System Interface (iSCSI)

List the documents published by Technical publications

 Comstar Administration Guide

List any recommended text books, external vendor documentation websites, training, CD and DVD courses

iSCSI: The Universal Storage Connection by John L. Hufferd (ISBN 020178419X).