KMDB and MDB, PMDB
Status 10/12/07
KMDB hasn't been paid alot of attention lately since we simply just didn't have the bandwidth to make it happen, it basically just builds. In lieu of that Guy focused on developing PMDB the "Poor Man's Debugger" and rightly named. Along with the Codewarrior JTAG debugger from Freescale Hardware Tools for the Target we grinded through it. To be honest lack of good tools were one of the biggest drags on our progress.
Status 06/12/07
PMDB updated to include a host of tools for debugging memory, and panics, traps.
Update 1/03/07 and a Happy New Year
MDB compiles now... but need to complete the link step. Our GCC for PPC points to GNU ld and we need to reconfigure it for our solaris ld.
12/15/06
First created ppc and chrp dirs in mdb and ppc in tools. Have used ppc32 and ppc64 for arch separation.
You may note in the early checkin history on this work there was a number of putbacks and deletions to i86 or sparc. This was due to a corrupted svn workspace that took us a while to figure out what was happening. We had to delete the mdb dir in total and then reconstruct.
Using make at cmb/mdb it will first attempt to build in the tools dir. These tools need to be x86 based since they run on the build machine. For the time being we copied setdynflag that was built on the std solaris vespa tree.
Some current issues
- missing msr.h in proto/root_ppc/usr/include, should be there with make install_h? copied it up on my ws for now
The problem in mdb/chrp related to the mdb/Makefile.module and building linktest KMDB_LINKTEST = is null for the moment.
PMDB Overview
PMDB = Poor Mans Debugger. Upon a panic we spit out the kernel stack trace and cpu specific reg info. It is a bit better then print's for everything.
There are two PMDB shells: 1) kernel configuration; 2) demo. They share the same interpreter, only the help text and set of commands are different. The interpreter is very primitive. It is RPN-style.
The rules are:
Words
--
Words are separated by whitespace.
Numbers
--
Anything starting with a digit is a hex number.
Numbers get pushed onto the argument stack.
Commands may may or may not pay attention to the argument stack.
It is possible to overflow the stack. Commands that take 1 boolean argument default to 1, if the argument stack is empty. In order to turn off a boolean you have to push 0 onto the stack. You need the space between words, even between words and numeric arguments. For example, "0 cbe" turns of CBE debugging, but "0cbe" is a hex number. kmf is not a boolean; you must give it a numeric argument.
Not all commands are safe all the time. For example, kmf can alter kmem debug
flags at any time. But, I think they were designed to be set once, early on.
The kernel memory allocator might get confused if the state of certain flags
changes during operations.
Startup options are meaningless but harmless after their time of applicability
has passed. For example, "ch" will not retroactively affect whether Solaris
interrupt handlers get installed, once the decision point has come and gone.
PMDB is not that sophisticated. If this were a GUI and PMDB were permanent,
then I render those commands that have become operationally vacuous in grey,
and disallow them, but this is quick and dirty.
The PMDB panic shell has two new commands: mod and vmem.
"mod" is the same as that in PMDB demo shell. It seems genuinely useful enough to keep.
"vmem" is new. It shows the heap_arena, text_arena, data_arena, and ctf_arena vmems.
The output is pretty detailed and raw. For each arena, it shows:
1) all spans,
2) all allocated segments,
3) all free segments.
I think it would be useful to show summaries of each arena:
1) total size of all spans,
2) total size of all allocated segments,
3) total size of all free segments.
I probably would have supported more PMDB demo shell commands in the panic shell, eventually. But, one the motivators for doing "mod" now is that I did "vmem" and noticed one huge module. It is "drv/ip".
KMDB Overview
(taken from M Simmons Log)
KMDB is a normal kernel module, and is loaded by krtld after unix, genunix, the CPU module, and the platmod have been loaded. This means that KMDB can't be used to debug early stages of krtld, but the advantages outweigh that cost. The bitness of the kernel is known at that point, so the corresponding version of kmdb can be loaded. The CPU module and platform modules are loaded, so KMDB can link against them, and can use functions that live in those modules. KMDB doesn't have to load anything, so it doesn't essentially have to be its own linker. Finally, the fact that it's a normal kernel module (actually two, but we won't get into that here) means that it can be relocated, and can thus be loaded well after boot by mdb -K (this is referred to as mod-loaded KMDB). Unfortunately, due to the way memory is allocated at the beginning of time, boot-loaded KMDB can't be unloaded, but mod-loaded KMDB can.
KMDB must be loadable both at boot and on-demand. This was accomplished by separating KMDB into two modules -- kmdbmod and kmdb.
Genassym and CTF
Generating assembly structure offset values with CTF
The Solaris kernel contains a fair amount of assembly, and this often needs to access C structures (and in particular know the size of such structures, and the byte offsets of their members). Since the assembler can't grok C, we need to provide constant values for it to use. This also applies to the C library and kmdb.
In the kernel, the header assym.h provides these values; for example:
These values are the byte offset of certain members into struct _kthread. For each of the types we want to reference from assembly, a template is provided in one of the offsets.in files. For the above, we can see in usr/src/uts/i86pc/ml/offsets.in:
This file contains structure names as well their members. Each of the members listed (which do not have to be in order, nor does the list need to be complete) cause a define to be generated; by default, an uppercase version of the member
name is used. As can be seen, this can be overridden by specifying a #define name to be used. The THREAD_SIZE define corresponds to the bytesize of the entire structure (it's also possible to generate a "shift" value, which is log2(size)).
To generate the header with the right offset and size values we need, a script is used to generate CTF data for the needed types, which then uses this data to output the assym.h header. This is a Perl script called genoffsets, and the build invokes it with a command line akin to:
genoffsets -s ctfstabs -r ctfconvert cc < offsets.in > assym.h
The hand-written offsets.in file serves as input to the script, and it generates the header we need. The script takes the following steps:
- Two temporary files are generated from the input. One is a C file consisting of #includes and any other pre-processor directives. The other contains the meat of the offsets file.
- The C file containing all the includes is built with the compile line given (I have stripped the compiler options above for readability).
- ctfconvert is run on the built .o file.
- The pre-processor is run across the second file (the temporary offsets file)
- This pre-processed file is passed to ctfstabs along with the .o file.
ctfstabs reads the input offsets file, and for each entry, looks up the relevant value in the CTF data contained in the .o file passed to it. It has two output modes (which I'll come to shortly), and in this case we are using the genassym driver to output the C header. As you can see,this is a fairly simple process of processing each line of the input and looking up the type data in the CTF contained in the .o file.
A similar process is used for generating forth debug files for use when debugging the kernel via the SPARC PROM. This takes a different format of offsets file more appropriate to generating the forth debug macros, described in the forth driver.
To finish off the output header, the output from a small program called genassym (or, on SPARC, genconst) is appended. It contains a bunch of printfs of constants. A lot of those don't actually need to be there since they're simple constant defines, and the assembly file could just include the right header, but others are still there for reasons such as:
- The macros which hide assembler syntax differences such as _MUL aren't implemented for the C compiler
- The value is an enum type, which ctfstabs doesn't support
- The constant is a complicated composed macro that the assembler can't grok
and other reasons. Whilst a lot of these could be cleaned up and removed from these files, it's probably not worth the development effort
except as a gradual change.