OpenSolaris

KSSL debugging

General

• check if kssl service is online

     svcs -a | grep kssl

• check if kssl kernel module is loaded

     modinfo | grep kssl

netstat

 Even if kssl service shows in svcs(1) output as online it is good idea to check listening ports. The output of the following netstat(1M) command for kssl service svc:/network/ssl/proxy:kssl-INADDR_ANY-443 and non-SSL enabled application running on port 8080 should be the following:

root:thanos:/# netstat -f inet -na -P tcp | grep LISTEN | egrep '(443)|(8080)'
192.29.75.65.443           *.*                0      0 49152      0 LISTEN

 In other words, only the SSL port configured in kssl tab should be visible for non-SSL enabled application.

kstat

 kssl module updates kstat(1M) counters for various events via KSSL_COUNTER macro. All kssl kstat counters can be displayed via following command:

thanos:~$ kstat -m kssl
module: kssl                            instance: 0     
name:   kssl_stats                      class:    crypto
        crtime                          82.1238848
        kssl_alloc_fails                0
        kssl_appdata_record_ins         5
        kssl_appdata_record_outs        2
        kssl_bad_pre_master_secret      0
        kssl_compute_mac_failure        0
        kssl_fallback_connections       0
        kssl_fatal_alerts               0
        kssl_full_handshakes            2
        kssl_no_suite_found             0
        kssl_proxy_fallback_failed      0
        kssl_record_decrypt_failure     0
        kssl_resumed_sessions           0
        kssl_sid_cache_hits             0
        kssl_sid_cache_lookups          0
        kssl_sid_uncached               0
        kssl_verify_mac_failure         0
        kssl_warning_alerts             0
        snaptime                        224345.0475506

 It is possible to watch the counters in real time by doing e.g. kstat -m kssl 1 which refreshes the display every second.

Example of kstat output interpretation

 If kssl_fallback_connections and kssl_proxy_fallback_failed counters are positive the this means a client is asking for an unsupported (by KSSL) cipher suite and the Apache web server is not configured to act as a fallback for that cipher suite. They should configure the Apache/mod_ssl SSL web server after running the ksslcfg command. Note that the certificate and key in httpd.conf/ssl.conf file should be the same as those passed to ksslcfg.

mdb

 All info about kssl proxies is stored in a table called kssl_entry_tab containing entries of type kssl_entry_t. Default size of the table is KSSL_TAB_INITSIZE which is 32. When more entries need to be added to the table the tables is grown. To dump all default entries in the tab use the following command in mdb -k:

> *kssl_entry_tab/32J
0x60001db8b00:  60001ec1b30     0               0               0
...

 To display one entry use the following command:

> 60001ec1b30::print kssl_entry_t
{
    ke_refcnt = 0x4
    ke_no_freeall = 0 (B_FALSE)
    ke_mutex = {
        _opaque = [ 0 ]
    }
    ke_laddr = 0
    ke_ssl_port = 0x1bb
    ke_proxy_port = 0x1f90
    sid_cache_timeout = 0x15180
    sid_cache_nentries = 0x1388
    sid_cache = 0x600020c0000
    kssl_cipherSuites = [ 0x5, 0x4, 0xa, 0x9, 0x2 ]
    kssl_cipherSuites_nentries = 0x5
    kssl_saved_Suites = [ 0x5, 0x4, 0xa, 0x9, 0x2 ]
    ke_private_key = 0x60001e47da0
    ke_server_certificate = 0x60001e5b9e0
    ke_cacert_chain = 0
    ke_proxy_head = 0x60001f05388
    ke_fallback_head = 0
}

 We can see what ports was this kssl service configured with:

> 60001ec1b30::print kssl_entry_t ke_ssl_port ke_proxy_port | =D
                443             8080            

dtrace

 KSSL can be debugged using the FBT and SDT probes. The SDT probes were added via RFE 6556447. Currently there are 80 SDT probes related to KSSL processing in the following functions in both ip and kssl kernel modules:

    MODULE:FUNCTION
    ip:tcp_kssl_input
    ip:tcp_kssl_input_callback
    ip:tcp_rcv_drain
    ip:tcp_rput_data
    kssl:kssl_compute_record_mac
    kssl:kssl_generate_ssl_ms
    kssl:kssl_generate_tls_ms
    kssl:kssl_get_next_record
    kssl:kssl_handle_any_record
    kssl:kssl_handle_client_hello
    kssl:kssl_handle_client_key_exchange
    kssl:kssl_handle_mblk
    kssl:kssl_handle_v2client_hello
    kssl:kssl_input
    kssl:kssl_mac_encrypt_record
    kssl:kssl_prov_evnt
    kssl:kssl_send_alert
    kssl:kssl_spec_init
    kssl:kssl_tls_PRF
    kssl:kssl_tls_P_hash

 To list all SDT KSSL probes use dtrace -l -n sdt:::kssl* The probes are named so that they share common kssl_ prefix which has several sub-prefixes, most notable sub-prefixes are mblk and err: * To list all error events in KSSL processing use the dtrace -l -n sdt:::kssl_err* command. * To list all probes which can dump mblk data use the dtrace -l -n sdt:::kssl_mblk* command. For more information about the rationale behind the probe placement and naming see the Adding dtrace SDT probes blog entry.