Software Debugging

There are two supported ways for debugging software with TEMU. Firstly the TEMU CLI supports assembler level debugging in itself. Secondly, TEMU is bundled with a GDB server, this server lets you start up a stand alone program (or run it from the CLI directly).

CLI Based Software Debugging

The TEMU command line interface will when reading ELF files, also load the symbol tables (there is also an API for inspecting ELF symbol tables). Thus, it is possible to disassemble named functions.

When disassembling code based on a function name or a virtual address, the disassembler prints special tokens indicating interesting addresses, such as the PC, nPC (for relevant targets) and trap table pointers. Only one token is printed, and the program counters will always have precedence over other tokens. These tokens are not printed when disassembling using physical addresses.

In addition to disassembling, it is possible to assemble instructions, and modify and inspect both registers and memory.

A global function can be disassembled using the func=name parameter, just give the name of the function to disassemble.

temu> dis cpu=cpu0 func=main
( pc) 40001934 040001934 9de3bf98 save %sp, 8088, %sp
(npc) 40001938 040001938 f027a044 st %i0, [%fp + 68]
      4000193c 04000193c f227a048 st %i1, [%fp + 72]
      40001940 040001940 c027bff8 st %g0, [%fp + 8184]
      40001944 040001944 82102001 or %g0, 1, %g1
      40001948 040001948 c227bffc st %g1, [%fp + 8188]
      4000194c 04000194c 82102000 or %g0, 0, %g1
      40001950 040001950 b0100001 or %g0, %g1, %i0
      40001954 040001954 81e80000 restore %g0, %g0, %g0
      40001958 040001958 81c3e008 jmpl %o7 + 8, %g0
      4000195c 04000195c 01000000 sethi 0, %g0

A local or static function can be disassembled by giving the function name prefixed with the file name and the scope resolution operator.

temu> dis cpu=cpu0 func=test.c::bar
      40001924 040001924 9de3bfa0 save %sp, 8096, %sp
      40001928 040001928 81e80000 restore %g0, %g0, %g0
      4000192c 04000192c 81c3e008 jmpl %o7 + 8, %g0
      40001930 040001930 01000000 sethi 0, %g0

Source Level Debugging

TEMU comes with built in source level debugging support, which is based on the DWARF debugging standard. This support is currently experimental and supports source listing, and symbolic and line based breakpoints. Since multiple applications with different DWARF data may be loaded at the same time (e.g. a boot loader and application, or multiple software partitions running under a hypervisor), the DWARF support is based around the notion of debugging contexts, which have been introduced in TEMU 2.2. Currently context management is manual and so is switching the active context. It is at present not possible to inspect symbolic data using the DWARF support.

There are at present three primary areas handled by the commands: context management (i.e. loading, unloading and switching the debugging context), source management (i.e. remapping paths due to moved source directories, and listing source code around the current address), break point handling (i.e. set and control break points). In addition to these, there are some DWARF specific commands that exist to support debugging of the source level debugging code.

The following commands exist at the moment, this explains the purpose of the commands:

experimental-debug-load-ctxt

Load an ELF file as a new debugging context.

experimental-debug-list-contexts

List all loaded debugging contexts.

experimental-debug-set-context

Set the current debugging context.

experimental-debug-dispose-ctxt

Remove debugging context.

experimental-debug-list-source

List source lines around the given address. Pass CPU argument to use the CPUs program counter. By default 5 lines is listed before and after.

experimental-debug-add-path

Add a path for searching for relatively named source files.

experimental-debug-remap-path

Add a remapping prefix. This is used to remap absolute paths to different directories. E.g. /home/foo/ to /home/bar/ will remap from one user dir to another one. This is particularly useful if the target software is built on a machine that is not the one you are debugging on.

experimental-debug-break

Add a breakpoint. Either at an address using the addr parameter or a named location using loc. Locations are in the form of LINENUMBER for locations in the current file (identified using the CPU argument). +NUMBER or -NUMBER for relative lines to the current one. FILE:LINENUMBER for explicit lines or FUNCNAME for a named function.

experimental-debug-mute-break

Prevent break point from printing message on a hit.

experimental-debug-demute-break

Ensure a break point prints a message on a hit.

experimental-debug-ignore-break

Ignore the break point (i.e. resume after hit, note that the message printing is controlled using the mute / un-mute commands), so it is easy to create a logging break point that does not stop the simulator.

experimental-debug-stop-break

Stop after the breakpoint has been hit. This is the default.

experimental-debug-simulate-break

Simulates a breakpoint hit at a given address. I.e. trigger the break point handler. This can be used to debug custom break point actions.

experimental-debug-list-cu

Print names of all compilation units in the current debugging context along with their starting address and implementation language.

experimental-debug-print-linenum-prog

Print the result of the line number program. This is DWARF specific and only useful for debugging line number and break location resolutions.

Legacy GDB Server

The legacy GDB server assumes it have full control over the running processors. It only works with separate CPUs and machine objects. For instructions relating to the scheduler compatible GDB server introduced in TEMU 4.1, please see New GDB Server.

TEMU (as of 2.1) comes with a built-in non-intrusive GDB server speaking the GDB remote protocol. The server is launched using the gdb-server command.

The server uses the SO_REUSEADDR socket option, so it is always possible to connect to the same port after disconnection without further delay.

To start the gdb-server inside the interactive CLI simply run the gdb-server command. It takes three parameters, machine or cpu, port and page-table-root.

When the server is running, GDB has control over the execution of the emulator, you can quit the GDB server command by interrupting it in the CLI (using Ctrl+C) or by disconnecting GDB.

If no arguments are given, the command defaults to machine0, cpu0 and port 6666 for the different arguments respectively. The page-table-root argument is only used if set explicitly.

The GDB server supports multi-core debugging, by exposing the cores as different threads. The multi-core support is limited in some ways, for example, breakpoints cannot be set per core.

Uploading binaries via the GDB remote protocol is very slow. It is recommended to load the binaries in the TEMU CLI and then specify which file you are debugging to GDB.
The GDB server does not know about operating system threads. Instead it treats a GDB-thread as a CPU core (numbering the threads from 1 for CPU 0 and up).
The GDB remote debugging protocol is not designed to be interfaced with emulators. One issue is that in order to inspect the stack, GDB will issue memory read commands to the remote target. This causes a problem in an emulator since many stack entries (especially on the SPARC target) will be in CPU registers. Thus, when the GDB program asks to read memory which is on the stack and in registers, the server will return the register content and not the memory content. Consequently if memory referring to register-shadowed memory content is modified, the remote target will write both memory and registers.

User Application Debugging

There are a number of problems when attempting to debug user applications. These stem from TEMU not being aware of the operating system it is debugging.

One impact of this, as mentioned before, is that the GDB server does not support inspection of OS threads.

Secondly, if an application runs with the MMU enabled, the default where the TEMU GDB server use the current processor’s MMU for address translation, may not work.

The MMU issues can be dealt with on some targets by setting the page-table-root argument to the gdb-server command.

This argument shall be the physical address, of the root page table pointer for the application in question.

Determining the root pointer can be difficult.

On the SPARC, the best way is to try to stop the emulator inside the application, and then inspecting the MMU state.

The page table root would then be calculated as:

Calculating the Page Table Root for the Current Application on a SPARC Processor
temu> hex((cpu0.mmuCtxtPtr << 4) + (cpu0.mmuCtxt * 4))

The above works for SPARC processors when the MMU is enabled.

If it is not possible stop when the application is running, it may be possible to use the page table walk mechanism to find the relevant pointer.

Experimentally Determining the Page Table Root
# Determine the current context
temu> cpu0.mmuCtxt
5

# Start with:
temu> cpu0.mmuCtxt = 0
temu> walk cpu=cpu0 addr=0x10000000
# If the walk fails pick the next mmuCtxt

temu> cpu0.mmuCtxt = 1
temu> walk cpu=cpu0 addr=0x10000000

# If the walk now passed we can calculate the page table root as:
temu> hex((cpu0.mmuCtxtPtr << 4) + (cpu0.mmuCtxt * 4))
'0x40001004'

# Restore the original mmuCtxt
temu> cpu0.mmuCtxt = 5

# Launch the GDB server using the computed page table root
temu> gdb-server cpu=cpu0 page-table-root=0x40001004
The above methods are all difficult to use in case the OS uses overlapping virtual address ranges for different applications. Contact your OS vendor for information on how to get the page table root if it is not practical using the methods above.

Example

Starting the GDB-server in the CLI:

temu> gdb-server machine=machine0 port=6666
Starting GDB server... (^C to stop)
GDB connected.

Connecting with GDB

(gdb) target remote localhost:6666
(gdb) file my-application.elf

New GDB Server

With the introduction of the new scheduler a new GDB server running in a separate thread was needed.

The new GDB server is available the plugin: GdbDebugger. To instantiate the GdbServer execute the following commands:

import GdbDebugger
GdbServer.new name=gdb port=6666 mem=mem0
connect-timesource obj=gdb ts=cpu0

The new command also allows for two additional arguments:

cpu

Processor model to limit breakpoints to this processor only.

debug

Set this to 1 to enable logging of GDB RSP protocol messages.

GDB is connected in the same way as detailed in Legacy GDB Server. When GDB is disconnected, the TEMU scheduler is automatically resumed.

Limitations

The GDB server cannot set breakpoints before a virtual to physical address mapping has materialized.

If two applications share the same virtual addresses, but different physical addresses. Then when setting a breakpoint, you will automatically set it on all matching addresses. This can be confusing as GDB has no way of determining which application is running. To limit this problem, it is possible to limit breakpoints to specific processors.

The limitations are due to GDB not being designed for systems level debugging, but focused on debugging single applications.