Software Debugging

There are two supported ways for debugging software with TEMU. Firstly the TEMU CLI supports assembler level debugging in itself. Secondly, TEMU is bundled with a GDB server, this server lets you start up a stand alone program (or run it from the CLI directly).

CLI Based Software Debugging

The TEMU command line interface will when reading ELF files, also load the symbol tables (there is also an API for inspecting ELF symbol tables). Thus, it is possible to disassemble named functions.

When disassembling code based on a function name or a virtual address, the disassembler prints special tokens indicating interesting addresses, such as the PC, nPC (for relevant targets) and trap table pointers. Only one token is printed, and the program counters will always have precedence over other tokens. These tokens are not printed when disassembling using physical addresses.

In addition to disassembling, it is possible to assemble instructions, and modify and inspect both registers and memory.

A global function can be disassembled using the func=name parameter, just give the name of the function to disassemble.

temu> dis cpu=cpu0 func=main
( pc) 40001934 040001934 9de3bf98 save %sp, 8088, %sp
(npc) 40001938 040001938 f027a044 st %i0, [%fp + 68]
      4000193c 04000193c f227a048 st %i1, [%fp + 72]
      40001940 040001940 c027bff8 st %g0, [%fp + 8184]
      40001944 040001944 82102001 or %g0, 1, %g1
      40001948 040001948 c227bffc st %g1, [%fp + 8188]
      4000194c 04000194c 82102000 or %g0, 0, %g1
      40001950 040001950 b0100001 or %g0, %g1, %i0
      40001954 040001954 81e80000 restore %g0, %g0, %g0
      40001958 040001958 81c3e008 jmpl %o7 + 8, %g0
      4000195c 04000195c 01000000 sethi 0, %g0

A local or static function can be disassembled by giving the function name prefixed with the file name and the scope resolution operator.

temu> dis cpu=cpu0 func=test.c::bar
      40001924 040001924 9de3bfa0 save %sp, 8096, %sp
      40001928 040001928 81e80000 restore %g0, %g0, %g0
      4000192c 04000192c 81c3e008 jmpl %o7 + 8, %g0
      40001930 040001930 01000000 sethi 0, %g0

Source Level Debugging

TEMU comes with built in source level debugging support, which is based on the DWARF debugging standard. This support is currently experimental and supports source listing, and symbolic and line based breakpoints. Since multiple applications with different DWARF data may be loaded at the same time (e.g. a boot loader and application, or multiple software partitions running under a hypervisor), the DWARF support is based around the notion of debugging contexts, which have been introduced in TEMU 2.2. Currently context management is manual and so is switching the active context. It is at present not possible to inspect symbolic data using the DWARF support.

There are at present three primary areas handled by the commands: context management (i.e. loading, unloading and switching the debugging context), source management (i.e. remapping paths due to moved source directories, and listing source code around the current address), break point handling (i.e. set and control break points). In addition to these, there are some DWARF specific commands that exist to support debugging of the source level debugging code.

The following commands exist at the moment, this explains the purpose of the commands:

experimental-debug-load-ctxt: Load an ELF file as a new debugging context.
experimental-debug-list-contexts: List all loaded debugging contexts.
experimental-debug-set-context: Set the current debugging context.
experimental-debug-dispose-ctxt: Remove debugging context.
experimental-debug-list-source: List source lines around the given address. Pass CPU argument to use the CPUs program counter. By default 5 lines is listed before and after.
experimental-debug-add-path: Add a path for searching for relatively named source files.
experimental-debug-remap-path: Add a remapping prefix. This is used to remap absolute paths to different directories. E.g. /home/foo/ to /home/bar/ will remap from one user dir to another one. This is particularly useful if the target software is built on a machine that is not the one you are debugging on.
experimental-debug-break: Add a breakpoint. Either at an address using the addr parameter or a named location using loc. Locations are in the form of LINENUMBER for locations in the current file (identified using the CPU argument). +NUMBER or -NUMBER for relative lines to the current one. FILE:LINENUMBER for explicit lines or FUNCNAME for a named function.
experimental-debug-mute-break: Prevent break point from printing message on a hit.
experimental-debug-demute-break: Ensure a break point prints a message on a hit.
experimental-debug-ignore-break: Ignore the break point (i.e. resume after hit, note that the message printing is controlled using the mute / un-mute commands), so it is easy to create a logging break point that does not stop the simulator.
experimental-debug-stop-break: Stop after the breakpoint has been hit. This is the default.
experimental-debug-simulate-break: Simulates a breakpoint hit at a given address. I.e. trigger the break point handler. This can be used to debug custom break point actions.
experimental-debug-list-cu: Print names of all compilation units in the current debugging context along with their starting address and implementation language.
experimental-debug-print-linenum-prog: Print the result of the line number program. This is DWARF specific and only useful for debugging line number and break location resolutions.

User Application Debugging

There are a number of problems when attempting to debug user applications. These stem from TEMU not being aware of the operating system it is debugging.

One impact of this, as mentioned before, is that the GDB server does not support inspection of OS threads.

Secondly, if an application runs with the MMU enabled, the default where the TEMU GDB server use the current processor’s MMU for address translation, may not work.

The MMU issues can be dealt with on some targets by setting the page-table-root argument to the gdb-server command.

This argument shall be the physical address, of the root page table pointer for the application in question.

Determining the root pointer can be difficult.

On the SPARC, the best way is to try to stop the emulator inside the application, and then inspecting the MMU state.

The page table root would then be calculated as:

Calculating the Page Table Root for the Current Application on a SPARC Processor

temu> hex((cpu0.mmuCtxtPtr << 4) + (cpu0.mmuCtxt * 4))

The above works for SPARC processors when the MMU is enabled.

If it is not possible stop when the application is running, it may be possible to use the page table walk mechanism to find the relevant pointer.

Experimentally Determining the Page Table Root

# Determine the current context
temu> cpu0.mmuCtxt
5

# Start with:
temu> cpu0.mmuCtxt = 0
temu> walk cpu=cpu0 addr=0x10000000
# If the walk fails pick the next mmuCtxt

temu> cpu0.mmuCtxt = 1
temu> walk cpu=cpu0 addr=0x10000000

# If the walk now passed we can calculate the page table root as:
temu> hex((cpu0.mmuCtxtPtr << 4) + (cpu0.mmuCtxt * 4))
'0x40001004'

# Restore the original mmuCtxt
temu> cpu0.mmuCtxt = 5

# Launch the GDB server using the computed page table root
temu> gdb-server cpu=cpu0 page-table-root=0x40001004

The above methods are all difficult to use in case the OS uses overlapping virtual address ranges for different applications. Contact your OS vendor for information on how to get the page table root if it is not practical using the methods above.

GDB Server

The GDB server is available the plugin: GdbDebugger. To instantiate the GdbServer execute the following commands:

import GdbDebugger
GdbServer.new name=gdb port=6666 mem=mem0
connect-timesource obj=gdb ts=cpu0

The new command also allows for two additional arguments:

cpu: Processor model to limit breakpoints to this processor only.
debug: Set this to 1 to enable logging of GDB RSP protocol messages.

GDB is connected in the same way as detailed in [_legacy_gdb_server]. When GDB is disconnected, the TEMU scheduler is automatically resumed.

Limitations

Virtual to Physical Address Translation

The GDB server cannot set breakpoints before a virtual to physical address mapping has materialized.

If two applications share the same virtual addresses, but different physical addresses. Then when setting a breakpoint, you will automatically set it on all matching addresses. This can be confusing as GDB has no way of determining which application is running. To limit this problem, it is possible to limit breakpoints to specific processors.

The limitations are due to GDB not being designed for systems level debugging, but focused on debugging single applications.

Re-Starting of GDB Server

In addition, the GDB server terminates after a disconnect is done. To start it again two options exist:

Start Server Thread Again

gdb.start-server

Delete and Create New Server

gdb.delete

GdbServer.new name=gdb port=6666 mem=mem0
connect-timesource obj=gdb ts=cpu0

Concurrent Usage

As the GDB server may be launched for different targets by either multiple users on the same system, or the same user, port assignment in a TEMU script may fail.

This could be an issue in the case a TEMU script is reused by multiple users.

Since TEMU scripts with failing commands either stop TEMU in batch mode, or return an error through the API.

End user applications must be prepared to deal with this.

Different options exists such as failing gracefully, or leaving it to the end user to start the server with a port number of choice.

Graceful Failure in TScript

if try GdbServer.new name=gdb mem=mem0 port=6666 {
  try connect-timesource obj=gdb ts=cpu0
  try gdb.start-server
}

Graceful Failure in C

if (temu_execCommandFile("gdb-script.temu")) {
  temu_printerr("Failed to create and start GDB server");
  gdbServerIsRunning = false;
}