Embedded Linux Debugging with JTAG ICE

1
Embedded Linux Debugging with JTAG ICE

• Kyoto Microcomputer Co., Ltd.
• http//www.kmckk.co.jp/eng

Translated by ikoma
2
Linux Debugging
3
Limitation of GDB(1)

• ptrace() system call in kernel is used for
  debugging. Debugging daemon(gdbserver) is one of
  user applications.

User Space
gdbserver
Target Application For Debugging
Kernel/Loadable Modules
ptrace
Driver
Kernel Space
4
Limitation of GDB (1)

• As kernel is one program, ptrace() can not run
  when breakpoint reached in device driver.

User Space
gdbserver
Target Application For Debugging
Breakpoint reached Kernel stops
Kernel/Loadable Modules
ptrace
Driver
Kernel Space
5
Limitation of GDB (1)

• While stopped in driver, it is not possible to
  peek into memory or to set break points on debug
  target app.

User Space
gdbserver
Target Application For Debugging
Kernel stops and gdbserver cannot work
Breakpoint reached Kernel stops
Kernel/Loadable Modules
ptrace
Driver
Kernel Space
6
Issue in GDB(2)

• Issue of debugging more than one processes at the
  same time

gdbserver
Process B
kernel
Process A
Break request
signal
execution stops
Break request
Process B may run
signal
execution stops
User Space
Kernel Space
7
Debugging Targets on Linux and Their Features
Type of Program Memory Space Address Paging Support for Debugging
Boot Loader NonMMU Space Fixed Address Nothing Same as Ordinary Embedded Debugging
Linux Kernel Non-multilplexed Kernel Space on MMU Fixed Address Nothing(1) Same as Ordinary Embedded Debugging
Loadable Module Non-multilplexed Kernel Space on MMU Relocatable On Demand Paging Requires Resolving Relocation and Support for Paging
Shared Library Logical Multiplexed Virtual Space on MMU Relocatable On Demand Paging Requires Resolving Relocation, Logical Multiplexed Virtual Space, and Support for Paging
Application Logical Multiplexed Virtual Space on MMU Fixed On Demand Paging Requires Logical Multiplexed Virtual Space and Support for Paging
1 Depending on CPU, Initialization Section may
be Paged Out
PARTNER-Jet Supports All Above
8
Technical Challenges

• Relocatable
• Actual addresses are unknown until kernel loads
  image onto memory.
• On Demand Paging
• When setting breakpoints or referring variables,
  it is not assured that the page is on memory.
• Multiplexed Virtual Spaces on MMU
• An address may designate different memory
  locations if the process IDs are different
• Support of hardware breakpoint
• Support of trace to process (ETM/AUD etc.)

9
Resolving Relocation(Loadable Modules)

• Tricks for Debugging on Loadable Modules
• Inserting software break before module
  initialization
• At insmod, execution of software break transfers
  control onto ICE. With that address, .text is
  resolved. .data and .bss are resolved using
  informoation embedded around SBP.
• As this method does not depend on Linux, the
  same method has also been applied for other
  systems using relocatable object.

define module_init(x) \ int init_module(void)
__attribute__((alias(x))) \ static inline
__init_module_func_t __init_module_inline(void)
\ return x define module_init(x) \
int init_module(void) \
__kmc_module_debug_start() return x() \
static inline __init_module_func_t
__init_module_inline(void) \ return x
asm(" .text") asm(" .long 0x4c434d81")
asm("__kmc_driver_start") asm(" .long
0xdeeedeee") asm(" .long 0x4c434d80")
asm(" .long __kmc_driver_name") asm("
.long __kmc_driver_tmp")
10
Resolving Relocation(Shared Library)

• PARNTER Obtains Info from Kernel
• Referring
• task_struct
• file_struct
• mm_struct
• of the process, /proc/(pid)/maps-like info is
  obtained.
• With the info obtained, .text, .data and .bss are
  relocated.

11
Resolving On Demand Paging

• Send program from ICE to cause read access at
  desired memory address whose page is not on
  memory
• Kernel loads the page
• User does not have to care about and PARTNER-Jet
  automatically does this operation
• PARTNER-Jet analyzes various info in kernel to
  make sure to access memory

12
Nikkei Electronics 2006/1/2 Issue KMCVirtualizing
Technologies to Support Debugging on Linux and
Multicore Environment P121
Kernel
Debugger Engine
User
Logical Address Space on MMU
Kernel loads pages of exec code
Debugger engine stops execution of process
User specify a break point on source code
Debugger engine let execute the code calling code
with specified break point
Page fault occurs and kernel loads page of new
exec code
Set the break point at the logical addr
corresponding to user specified location
User orders to resume process
Debugger engine restarts the process
Kernel executes the process to the breakpoint
Debugger engine stops the process at the
breakpoint
Fig.5 Intentional Page Fault Linux kernel does
not load all execution code onto main memory at
once. Page fault, detecting that code to
execute has not been loaded on memory,causes the
execution code to be read into. In order to make
it possible to set breakpoint at point kernel has
not yet loaded on memory, we have intentionally
made page fault occur. We have implemented the
function to call beforehand execution code, which
correspondis to the user specified breakpoint.
13
Resolving Multiplexed Memory Spaces

• ICE Manages Virtual Spaces
• While debugging process, tracks the management
  structure of the process(task_struct)
• When breakpoint reached, keeps the state of the
  process at that timing
• Minimal Kernel Patches
• Essential to dispatch process of real time
  trace(sched.c). And some others.

void __kmc_schedule(prev,next) struct task_struct
prev, next int index_next
index_next__kmc_schedules_index
(KMC_MAX_SHEDULE_LIST-1)
__kmc_schedules_index
__kmc_schedules_list_pidindex_nextnext-gtpid
ifdef CONFIG_KMC_TRACE_EXT
__kmc_schedule_call(prev,next) endif
/ This just switches the register
state and the stack. /
switch_to(prev, next, prev)
__schedule_tail(prev)
14
Demonstration
15
Tricks for Linux

• Attaching to Running Process
• Possible at any timing, while running, while
  blocking in kernel, or while bloking in process
• ICE saves PC which has been saved as context, and
  modifies it to address in attach area. When
  rescheduled, attach area is executed and breaks.
• Automatic Loading of Debugging Information
• Loading automattically all debugging information
  of shared libraries which the process is using,
  and relocating automatically
• Analyzing the control structure of the process
• Function Trace for Process
• Even when stopped in kerenl space, backtrace of
  the specified thread can be referred
• ICE refers to the PC, SP, FP and stacks saved as
  context and replays
• Application Mode(See following slides)
• Debugging while keeping other processes than
  target process alive

16
Application Mode - Virtual ICE Technology -

• ICE Dedicated to Process
• Can debug only processes on virtual spaces with
  external ICE
• Memory swapping is fully supported
• Debugging of Target Process Only
• Debuggable with other processes than target and
  kernel to keep running
• Effective Use of ICE Functions
• Hardware breakpoints, traces

17
Application Mode - Virtual ICE Technology -
?????????
Process1
Process2
Process3
Device Drivers
Operating System
CPU
Target System
JTAG
ICE PARTNER-Jet
PARTNER Debug Control DLL
Debugger PARTNER/Win
VirtualICE
VirtualICE
VirtualICE
Debugger PARTNER/Win
Debugger PARTNER/Win
Debugger PARTNER/Win
Debugging Host PC
Debugging of kernel modules
Debugging of Process1
Debugging of Process2
Debugging of Process3
18
Profiling Features for Linux

• Execute Profile from JTAG ICE
• No target memory used to keep trace data (for
  memory-tight embedded systems)
• Implemented with Minimal Patches to Kernel
• 4 patches in 3 source file for ARM11
• Efficient communication by using ARM DCC

static inline unsigned int jtag_com_stat11(void)
unsigned int stat __asm__("mrc
p14,0,0,c0,c1""r"(stat)) return
stat static inline void jtag_com_put11(unsigne
d int data) __asm__("mcr p14,0,0,c0,c5""r
"(data)) return
19
Profiling for Linux

• Key Features
• Profilable integrating both kernel and processes
• At 100usec sampling, load on target is 1/100 or
  less (in case of ARM9)
• Get transitions of generation/distruction/switchin
  g of processes and threads, displaying with
  process IDs/process names
• Get execution time ratio per process
• Get execution time ratio of each function in
  processes/kernel
• Get execution time ratio of processes/kernel
• Referring on debugger, as well as saving data in
  CSV format to analyze yourself withEXCEL etc.

20
Being Open Source
21
Efficient Debugging on Embedded Linux

• Fully Enjoy Being Open Source
• Debugging environment becomes more powerful with
  combination of ICE kernel modification, than
  with ICE only
• It is easier for us, tool vendor, to develop
  tools for Linux because of open source
• PARNTER-Jet is monitoring major data structures
  in Linux kernel(task_struct,mm_struct,file_struct
  etc. )
• Matching of trace of ETM/AUD etc. and souce code
  enables to investigate in detail of Linux dynamics

22
Interaction of Linux and Tools

• Transfers trace and logging data from kernel and
  processes through JTAG
• Memory freeze, trace freeze, and analysis
  functions at specified location in kernel

Analyzing App
App A
App B
App C
JTAG ICE
Kernel
Drivers
23

• Contact
• Headquarter
• 2-44 Ooe-nakayama-cho, Nishikyo-ku, Kyoto,
  610-1104, Japan
• TEL [phone] FAX [phone]
• Tokyo Office
• R Bldg 5F, 2-14-4 Shimbashi, Minato-ku, Tokyo,
  105-0004, Japan
• TEL [phone] FAX [phone]
• URLhttp//www.kmckk.co.jp/eng
• jp-info_at_kmckk.co.jp
• er-info_at_kmck.co.jp