iSER Debugging Tips

iSER Debugging Tips

MDB commands

 You can use a a number of MDB commands to display useful debugging information.

 First, load the iSCSI MDB module:

{{{
> ::load idm
}}}

 List iSCSI targets with associated sessions and connections:

{{{
> ::iscsi_tgt -sc
}}}

 List iSCSI connections with associated active tasks and buffers:

{{{
> ::iscsi_conn -tb
}}}

 List iSCSI connections and tasks with verbose output:

{{{
> ::iscsi_conn -tv
}}}

 There are lots of other combinations. If things hang you can get a good diagnostic snapshot of the iscsit state with:

{{{
> ::iscsi_tgt -sctbv
}}}

 

Catching Memory Usage bugs

 You can add the line "set kmem_flags=0xf" (if it's not already set) in /etc/system. It will catch a lot of copy memory usage errors and it will catch them early enough that they are usually easy to find.

Reporting Transport Type

 Right now we don't have a CLI mechanism to report the underlying transport type(iSCSI or iSER) of a connection. You can use MDB to determine this information though:

1. Start MDB

 2. Make sure iscsi plugin is loaded:

{{{
> ::load iscsi
}}}

 

 3. Get list of connections:

{{{
> ::iscsi_conn
}}}

 4. For each connection in the list print the ic_transport_t field using the following command:

{{{
> ffffff04ed620880 ::print idm_conn_t ic_transport_type
ic_transport_type = 0 (IDM_TRANSPORT_TYPE_ISER)
> ffffff04ebd58b00 ::print idm_conn_t ic_transport_type
ic_transport_type = 0 (IDM_TRANSPORT_TYPE_ISER)
}}}

 

Examining hold/rele reference using MDB

 iSER uses refcnts on a number of objects to ensure that they are not deleted while still being used. If references are not being maintained properly, you can usually get some hint about what's going wrong with the references by looking at the audit data with MDB. Some of the dcmds have a -R option to print the reference audit data for the object:

{{{
> ::iscsi_conn -R
> ::iscsi_tgt -R
> ::iscsi_task -R
}}}

 

 Or you can look at the object from any refcnt object using addr::iscsi_refcnt.

 Example output (this is equivalent to addr::iscsi_conn -R):

{{{
> ffffff4689186000 ::print -a idm_conn_t ic_refcnt  {
   ffffff4689186018 ic_refcnt.ir_refcnt = 0x1
[snip]
> ffffff4689186018 ::iscsi_refcnt   
RefCnt: 1      idm_refcnt_rele+0x5e
               idm_conn_rele+0x16
               idm_task_free+0x6b
               iscsit_task_free+0x1c
               iscsit_lport_task_free+0x35
               stmf_task_free+0x56
               stmf_worker_task+0x2d6
               thread_start+0x8
RefCnt: 2      idm_refcnt_rele+0x5e
               idm_conn_rele+0x16
               idm_conn_event_handler+0x1c7
               taskq_thread+0x194
               thread_start+0x8
RefCnt: 3      idm_refcnt_async_wait_ref+0x73
               idm_update_state+0x1b5
               idm_state_s8_cleanup+0x68
               idm_conn_event_handler+0xf9
               taskq_thread+0x194
               thread_start+0x8
RefCnt: 3      idm_refcnt_hold+0x5e
      idm_conn_hold+0x16
               idm_conn_event_locked+0x67
               idm_conn_event+0x48
               idm_cleanup_timeout+0x1a
               callout_execute+0xc1
               taskq_thread+0x194
               thread_start+0x8
RefCnt: 2      idm_refcnt_rele+0x5e
               idm_conn_rele+0x16
               idm_conn_event_handler+0x1c7
               taskq_thread+0x194
               thread_start+0x8
RefCnt: 3      idm_refcnt_rele+0x5e
               idm_conn_rele+0x16
               idm_sotx_thread+0x258
               thread_start+0x8
RefCnt: 4      idm_refcnt_rele+0x5e
               idm_conn_rele+0x16
               idm_sorx_thread+0x1bb
               thread_start+0x8    
RefCnt: 5      idm_refcnt_hold+0x5e
               idm_conn_hold+0x16
               idm_conn_event_locked+0x67
               idm_conn_event+0x48
               idm_sorx_thread+0x1b3
               thread_start+0x8
}}}