Linux PCMCIA HOWTO: Debugging tips and programming information

Next Previous Contents

## 7. Debugging tips and programming information

## 7.1 Submitting useful problem reports

The best way to submit reports is to use the online pcmcia-cs forums or the bug tracker at SourceForge. That way, other people can see current problems (and fixes or workarounds, if available). Here are some things that should be included in all bug reports:

- Your system brand and model.
- All PCMCIA card(s) you are using.
- Your Linux kernel version (i.e., ```uname -rv`''), and PCMCIA driver version (i.e., ```cardctl -V`'').
- Output of 'lspci -v'
- Any changes you have made to the startup files in `/etc/pcmcia`, or to the PCMCIA startup script.
- All PCMCIA-related messages in your system log file. That includes startup messages, and messages generated when your cards are configured.

All the PCMCIA modules and the `cardmgr` daemon send status messages to the system log. These will usually end up somewhere like `/var/log/messages` or `/var/log/daemon.log`. These files should be the first place to look when tracking down a problem. When submitting a bug report, always include the relevant contents of these files. If you are having trouble finding your system messages, check `/etc/syslog.conf` to see how different classes of messages are handled.

Before submitting a bug report, please check to make sure that you are using an up-to-date copy of the driver package. While it is somewhat gratifying to read bug reports for things I've already fixed, it isn't a particularly constructive use of my time.

If you do not have web access, bug reports can be sent to me at `dahinds@users.sourceforge.net`. However, I prefer that bug reports be posted to the pcmcia-cs SourceForge site, so that they can be seen by others.

## 7.2 Interpreting kernel trap reports

If your problem involves a kernel fault, the register dump from the fault is only useful if you can translate the fault address, EIP, to something meaningful. Recent versions of `klogd` attempt to translate fault addresses based on the current kernel symbol map, but this may not work if the fault is in a module, or if the problem is severe enough that `klogd` cannot finish writing the fault information to the system log.

If a fault is in the main kernel, the fault address can be looked up in the `System.map` file. This may be installed in `/System.map` or `/boot/System.map`. If a fault is in a module, the `nm` command gives the same information, however, the fault address has to be adjusted based on the module's load address. Let's say that you have the following kernel fault:

> `

>     Unable to handle kernel NULL pointer dereference
>     current->tss.cr3 = 014c9000, %cr3 = 014c9000
>     *pde = 00000000
>     Oops: 0002
>     CPU:    0
>     EIP:    0010:[<c2026081>]
>     EFLAGS: 00010282

> `

The fault address is 0xc2026081. Looking at `System.map`, we see that this is past the end of the kernel, i.e., is in a kernel module. To determine which module, check the output of ```ksyms -m | sort`'':

> `

>     Address   Symbol                            Defined by
>     c200d000  (35k)                             [pcmcia_core]
>     c200d10c  register_ss_entry                 [pcmcia_core]
>     c200d230  unregister_ss_entry               [pcmcia_core]
>               ...
>     c2026000  (9k)                              [3c574_cs]
>     c202a000  (4k)                              [serial_cs]

> `

So, 0xc2026081 is in the `3c574_cs` module, and is at an offset of 0x0081 from the start of the module. We cannot look up this offset in `3c574_cs.o` yet: when the kernel loads a module, it inserts a header at the module load address, so the real start of the module is offset from the address shown in `ksyms`. The size of the header varies with kernel version: to find out the size for your kernel, check a module that exports symbols (like `pcmcia_core` above), and compare a symbol address with `nm` output for that symbol. In this example, `register_ss_entry` is loaded at an offset of 0xc200d10c - 0xc200d000 = 0x010c, while ```nm pcmcia_core.o`'' shows the offset as 0x00c0, so the header size is 0x010c - 0x00c0 = 0x004c bytes.

Back to `3c574_cs`, our fault offset is 0x0081, and subtracting the 0x004c header, the real module offset is 0x0035. Now looking at ```nm 3c574_cs.o | sort`'', we see:

> `

>     0000002c d if_names
>     0000002c t tc574_attach
>     00000040 d mii_preamble_required
>     00000041 d dev_info

> `

So, the fault is located in `tc574_attach()`.

In this example, the fault did not cause a total system lockup, so `ksyms` could be executed after the fault happened. In other cases, you may have to infer the module load addresses indirectly. The same sequence of events will normally load modules in the same order and at the same addresses. If a fault happens when a certain card is inserted, get the `ksyms` output before inserting the card, or with a different card inserted. You can also manually load the card's driver modules with `insmod` and run `ksyms` before inserting the card.

For more background, see ```man insmod`'', ```man ksyms`'', and ```man klogd`''. In the kernel source tree, `Documentation/oops-tracing.txt` is also relevant. Here are a few more kernel debugging hints:

- Depending on the fault, it may also be useful to translate addresses in the ``Call Trace'', using the same procedure as for the main fault address.
- To diagnose a silent lock-up, try to provoke the problem with X disabled, since kernel messages sent to the text console will not be visible under X.
- If you kill `klogd`, most kernel messages will be echoed directly on the text console, which may be helpful if the problem prevents `klogd` from writing to the system log.
- To cause all kernel messages to be sent to the console, for 2.2 and later kernels, if `/proc/sys/kernel/printk` exists, do:

  > `

  >     echo 8 > /proc/sys/kernel/printk

  > `

- The key combination <RightAlt><ScrLk> prints a register dump on the text console. This may work even if the system is otherwise completely unresponsive, and the EIP address can be interpreted as for a kernel fault.
- For 2.2 and later kernels configured with `CONFIG_MAGIC_SYSRQ` enabled, various emergency functions are available via special <Alt><SysRq> key combinations, documented in `Documentation/sysrq.txt` in the kernel source tree.

## 7.3 Low level PCMCIA debugging aids

The PCMCIA modules contain a lot of conditionally-compiled debugging code. Most of this code is under control of the `PCMCIA_DEBUG` preprocessor define. If this is undefined, debugging code will not be compiled. If set to 0, the code is compiled but inactive. Larger numbers specify increasing levels of verbosity. Each module built with `PCMCIA_DEBUG` defined will have an integer parameter, `pc_debug`, that controls the verbosity of its output. This can be adjusted when the module is loaded, so output can be controlled on a per-module basis without recompiling.

Your default configuration for `syslogd` may discard kernel debugging messages. To ensure that they are recorded, edit `/etc/syslog.conf` to ensure that ```kern.debug`'' messages are recorded somewhere. See ```man syslog.conf`'' for details.

There are a few register-level debugging tools in the `debug_tools/` subdirectory of the PCMCIA distribution. The `dump_tcic` and `dump_i365` utilities generate register dumps for ISA-to-PCMCIA controllers. In 3.1.15 and later releases, `dump_i365` is replaced by `dump_exca`, which is similar but also works for PCI-to-CardBus bridges. Also new in 3.1.15 for CardBus bridges is the `dump_cardbus` tool, which interprets the CardBus-specific registers. These are all most useful if you have access to a datasheet for the corresponding controller chip. The `dump_cis` utility (`dump_tuples` in pre-3.0.2 distributions) lists the contents of a card's CIS (Card Information Structure), and decodes most of the important bits. And the `dump_cisreg` utility displays a card's local configuration registers.

The `memory_cs` memory card driver is also sometimes useful for debugging problems with 16-bit PC Cards. It can be bound to any card, and does not interfere with other drivers. It can be used to directly access any card's attribute memory or common memory. Similarly for CardBus cards, the `memory_cb` driver can be bound to any 32-bit card, to give direct access to that card's address spaces. See the man pages for more information.

## 7.4 /proc/bus/pccard

On 2.2 and later kernels, the PCMCIA package will create a tree of status information under `/proc/bus/pccard`. Much of the information can only be interpreted using the data sheets for the PCMCIA host controller. Its contents may depend on how the drivers were configured, but may include all or some of the following:

`/proc/bus/pccard/{irq,ioport,memory}`

  If present, these files contain resource allocation information to supplement the normal kernel resource tables. Recent versions of the PCMCIA system may obtain additional resource information from the Plug and Play BIOS if configured to do so.

`/proc/bus/pccard/drivers`

  In recent releases, this lists all currently loaded PCMCIA client drivers. Unlike `/proc/modules`, it also lists drivers that may be statically linked into the kernel.

`/proc/bus/pccard/*/info`

  For each socket, describes that socket's host controller and its capabilities.

`/proc/bus/pccard/*/exca`

  This contains a dump of a controller's ``ExCA'' Intel i82365sl-compatible register set.

`/proc/bus/pccard/*/{pci,cardbus}`

  For CardBus bridges, a dump of the bridge's PCI configuration space, and a dump of the bridge's CardBus configuration registers.

## 7.5 Writing Card Services drivers for new cards

The Linux PCMCIA Programmer's Guide is the best documentation for the client driver interface. The latest version is always available from `projects.sourceforge.net` in `/pub/pcmcia-cs/doc`, or on the web at http://pcmcia-cs.sourceforge.net.

For devices that are close relatives of normal ISA devices, you will probably be able to use parts of existing Linux drivers. In some cases, the biggest stumbling block will be modifying an existing driver so that it can handle adding and removing devices after boot time. Of the current drivers, the memory card driver is the only ``self-contained'' driver that does not depend on other parts of the Linux kernel to do most of the dirty work.

In many cases, the largest barrier to supporting a new card type is obtaining technical information from the manufacturer. It may be difficult to figure out who to ask, or to explain exactly what information is needed. However, with a few exceptions, it is very difficult if not impossible to implement a driver for a card without technical information from the manufacturer.

I have written a dummy driver with lots of comments that explains a lot of how a driver communicates with Card Services; you will find this in the PCMCIA source distribution in `clients/dummy_cs.c`.

## 7.6 Guidelines for PCMCIA client driver authors

I have decided that it is not really feasible for me to distribute all PCMCIA client drivers as part of the PCMCIA package. Each new driver makes the main package incrementally harder to maintain, and including a driver inevitably transfers some of the maintenance work from the driver author to me. Instead, I will decide on a case by case basis whether or not to include contributed drivers, based on user demand as well as maintainability. For drivers not included in the core package, I suggest that driver authors adopt the following scheme for packaging their drivers for distribution.

Driver files should be arranged in the same directory scheme used in the PCMCIA source distribution, so that the driver can be unpacked on top of a complete PCMCIA source tree. A driver should include source files (in `./modules/`), a man page (in `./man/`), and configuration files (in `./etc/`). The top level directory should also include a README file.

The top-level directory should include a makefile, set up so that ```make -f ...` all'' and ```make -f ... install`'' compile the driver and install all appropriate files. If this makefile is given an extension of `.mk`, then it will automatically be invoked by the top-level `Makefile` for the `all` and `install` targets. Here is an example of how such a makefile could be constructed:

> `

>     # Sample Makefile for contributed client driver
>     FILES = sample_cs.mk README.sample_cs \
>             modules/sample_cs.c modules/sample_cs.h \
>             etc/sample.conf etc/sample etc/sample.opts \
>             man/sample_cs.4
>     all:
>             $(MAKE) -C modules MODULES=sample_cs.o
>     install:
>             $(MAKE) -C modules install-modules MODULES=sample_cs.o
>             $(MAKE) -C etc install-clients CLIENTS=sample
>             $(MAKE) -C man install-man4 MAN4=sample_cs.4
>     dist:
>             tar czvf sample_cs.tar.gz $(FILES)

> `

This makefile uses install targets defined in 2.9.10 and later versions of the PCMCIA package. This makefile also includes a ``dist'' target for the convenience of the driver author. You would probably want to add a version number to the final package filename (for example, `sample_cs-1.5.tar.gz`). A complete distribution could look like:

> `

>     sample_cs.mk
>     README.sample_cs
>     modules/sample_cs.c
>     modules/sample_cs.h
>     etc/sample.conf
>     etc/sample
>     etc/sample.opts
>     man/sample_cs.4

> `

With this arrangement, when the contributed driver is unpacked, it becomes essentially part of the PCMCIA source tree. It can make use of the PCMCIA header files, as well as the machinery for checking the user's system configuration, and automatic dependency checking, just like a ``normal'' client driver.

In this example, `etc/sample` and `etc/sample.opts` would be the new driver's configuration scripts (if needed), and `etc/sample.conf` would contain any additions to the PCMCIA card configuration file. Starting with the 3.1.6 release, `cardmgr` will automatically process any `*.conf` files installed in `/etc/pcmcia`, so installation of contributed drivers should no longer require hand editing configuration files.

I will accept client drivers prepared according to this specification and place them in the `/pub/pcmcia-cs/contrib` directory on `projects.sourceforge.net`. The README in this directory will describe how to unpack a contributed driver.

The client driver interface has not changed much over time, and has almost always preserved backwards compatibility. A client driver will not normally need to be updated for minor revisions in the main package. I will try to notify authors of contributed drivers of changes that require updates to their drivers.

## 7.7 Guidelines for Linux distribution maintainers

If your distribution has system configuration tools that you would like to be PCMCIA-aware, please use the `*.opts` files in `/etc/pcmcia` for your ``hooks.'' These files will not be modified if a user compiles and installs a new release of the PCMCIA package. If you modify the main configuration scripts, then a fresh install will silently overwrite your custom scripts and break the connection with your configuration tools. Contact me if you are not sure how to write an appropriate option script, or if you need additional capabilities.

It is helpful for users (and for me) if you can document how your distribution deviates from the PCMCIA package as described in this document. In particular, please document changes to the startup script and configuration scripts. If you send me the appropriate information, I will include it in the Notes about specific Linux distributions.

When building PCMCIA for distribution, consider including contributed drivers that are not part of the main PCMCIA package. For reasons of maintainability, I am trying to limit the core package size, by only adding new drivers if I think they are of particularly broad interest. Other drivers will be distributed separately, as described in the previous section. The split between integral and separate drivers is somewhat arbitrary and partly historical, and should not imply a difference in quality.

Next Previous Contents