Debugging process in linux

Linux Applications Debugging Techniques/The debugger

Contents

Preparations [ edit | edit source ]

Someday some hard to reproduce issue will be found on a production machine. Typically, such a machine is difficult to access, has no development environment and nothing can be installed on it. At most it will have gdb, but very likely not. Typically also, the heaviest user will be the one to find the issue and typically still, the heaviest user is the one being the bigger — money-wise. And that issue has to be root cause diagnosed and fixed.

Unless the application has been prepared for this forensic gathering moment, not much can be done to diagnose where the issue is. Thus, preparations should start with compilation:

• Have a «symbol server» and make sure it is reacheable. Compile on the symbol server.
• Compile the release with debugging symbols. Strip them if you do not want to ship them but keep them.
• Ship gdbserver with the application for remote debugging.

These preparations will allow one to:

• Debug the application running on any machine, including machines where there is no gdb installed.
• Debug from any machine that has network visibility to the symbol server.

Also, think beforehand how would you debug on the machine:

• Embed a breakpoint in the code, at hard of reach places of interest, then
  • Start the application
  • Attach to it with the debugger
  • Wait until the breakpoint is hit

The «symbol server» [ edit | edit source ]

One way to easily reach the right code from within the debugger is to build the binaries within an auto-mounted folder, each build in its own sub-folder. The same auto-mount share should be accessible from the machine you are debugging on.

Debian [ edit | edit source ]

• Install autofs

• In /etc/auto.master uncomment the line:

• In /etc/exports export the folder:

• As root: restart autofs & nfs and export the build share:

Redhat [ edit | edit source ]

• Export the folder: edit /etc/exports
• As root (RedHat): service autofs start

Finally, build the binaries within the automounted directory on the build machine (here the build machine is bear):

Notice the filename path that is resolved with the symbol information:

If the symbols have been stripped from the binaries, point gdb to the folders where the symbols are with the debug-file-directory directive.

References [ edit | edit source ]

Source Code [ edit | edit source ]

To point the debugger to the source files:

Remote debugging [ edit | edit source ]

• On the machine where the application runs (appmachine):
  • If gdbserver is not present, copy it over.
  • Start the application.
  • Start gdbserver: gdbserver gdbmachine:2345 --attach program
• On gdbmachine:
  • At the gdb prompt, enter: target remote appmachine:2345

Sometimes you may have to tunnel over ssh:

• On gdbmachine:
  • ssh -L 5432:appmachine:2345 user@appmachine
  • At the gdb prompt: target remote localhost:5432

References [ edit | edit source ]

Attaching to a process [ edit | edit source ]

Find out the PID of the process, then:

Debugging programs that spawn multiple children [ edit | edit source ]

• set detach-on-fork off
• see «all-stop» vs «non-stop» modes in the GDB documentation and their related settings

Embedding breakpoints in the source [ edit | edit source ]

On x86 platforms:

Or a more elaborate one:

This will break into the debugger on hard to reach conditions:

References [ edit | edit source ]

Data breakpoints (watchpoints) [ edit | edit source ]

Watchpoints can be implemented either in software either in hardware if the CPU supports it. Typically on an Intel processor there are eight debug registers out of which only four can be used for hardware breakpoints and this limits the number of watchpoints system wide.

Breakpoints conditional on caller [ edit | edit source ]

This requires gdb 7.9 or later, configured with python support:

The text user interface [ edit | edit source ]

GDB features a text user interface for code, disassembler and registers. For instance:

• Ctrl-x 1 will show the code pane
• Ctrl-x a will hide the TUI panes

None of the GUI interfaces to gdb (Qt Creator stands out for being intuitive and easy to use) can offer access to all of the gdb functionality.

curses gdb offers an improved TUI. A comprehensive list of debugger GUIs is available here.

Reverse debugging [ edit | edit source ]

As an example, reverse debugging is a functionality no GUI offers access to:

Register watch [ edit | edit source ]

You can watch registers. Note this will force the debugger to single step the debugged program and it will run very slowly:

References [ edit | edit source ]

.gdbinit [ edit | edit source ]

As a note, in upcoming gdb releases, .gdbinit will be replaced by gdb-gdb.gdb:

C++ support [ edit | edit source ]

Canned gdb macros [ edit | edit source ]

Mangling [ edit | edit source ]

gdb might need a bit of guidance with C++11 binaries:

Templates [ edit | edit source ]

One can eventually use templight to debug and profile templates.

Источник

10 Strace Commands for Troubleshooting and Debugging Linux Processes

strace is a powerful command line tool for debugging and trouble shooting programs in Unix-like operating systems such as Linux. It captures and records all system calls made by a process and the signals received by the process.

It displays the name of each system call together with its arguments enclosed in a parenthesis and its return value to standard error; you can optionally redirect it to a file as well.

In this article, we will explain 10 strace command examples for troubleshooting and debugging programs and processes in a Linux system.

How to Install Strace Process Monitoring Tool in Linux

If strace is not pre-installed on your Linux system, run the appropriate command below for your distribution, to install it.

In case a program crashes or behaves in a way not expected, you can go through its systems calls to get a clue of what exactly happened during its execution. As we will see later on, system calls can be categorized under different events: those relating to process management, those that take a file as an argument, those that involve networking, memory mapping, signals, IPC and also file descriptor related system calls.

You can either run a program/command with strace or pass a PID to it using the -p option as in the following examples.

1. Trace Linux Command System Calls

You can simply run a command with strace like this, here we are tracing of all system calls made by the df command.

>P\t». 832) = 832 fstat(3, ) = 0 mmap(NULL, 3971488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f82f7310000 .

From the output above, you can see various types of system calls made by df command, for example.

• open – is the type of system call
• (“/etc/ld.so.cache”, O_RDONLY|O_CLOEXEC) – system call argument
• 3 – system call return value

Below is an sample output showing the write system calls, that displays df command output on the screen.

2. Trace Linux Process PID

If a process is already running, you can trace it by simply passing its PID as follows; this will fill your screen with continues output that shows system calls being made by the process, to end it, press [Ctrl + C] .

%», 4096>], msg_controllen=0, msg_flags=0>, 0) = 32 recvmsg(4, 0x7ffee4dbf870, 0) = -1 EAGAIN (Resource temporarily unavailable) recvmsg(4, 0x7ffee4dbf850, 0) = -1 EAGAIN (Resource temporarily unavailable) poll([, , , , , ], 6, -1) = 1 ([]) read(31, «2», 1) = 1 recvmsg(4, 0x7ffee4dbf850, 0) = -1 EAGAIN (Resource temporarily unavailable) poll([, , , , , ], 6, 0) = 1 ([]) read(31, «2», 1) = 1 recvmsg(4, 0x7ffee4dbf850, 0) = -1 EAGAIN (Resource temporarily unavailable) poll([, , , , , ], 6, 0) = 0 (Timeout) mprotect(0x207faa20000, 8192, PROT_READ|PROT_WRITE) = 0 mprotect(0x207faa20000, 8192, PROT_READ|PROT_EXEC) = 0 mprotect(0x207faa21000, 4096, PROT_READ|PROT_WRITE) = 0 mprotect(0x207faa21000, 4096, PROT_READ|PROT_EXEC) = 0 .

3. Get Summary of Linux Process

Using the -c flag, you can generate a report of total time, calls, and errors for each system call, as follows.

4. Print Instruction Pointer During System Call

The -i option displays the instruction pointer at the time of each system call made by the program.

>P\t». 832) = 832 [00007faf9cafb2b4] fstat(3, ) = 0 [00007faf9cafb47a] mmap(NULL, 3971488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7faf9c716000 [00007faf9cafb517] mprotect(0x7faf9c8d6000, 2097152, PROT_NONE) = 0 .

5. Show Time of Day For Each Trace Output Line

You can also print the time of day for each line in the trace output, by passing the -t flag.

>P\t». 832) = 832 15:19:25 fstat(3, ) = 0 15:19:25 mmap(NULL, 3971488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f8c7ebec000 15:19:25 mprotect(0x7f8c7edac000, 2097152, PROT_NONE) = 0 .

6. Print Command Time Spent in System Calls

To shows the time difference between the starting and the end of each system call made by a program, use the -T option.

7. Trace Only Specific System Calls

In the command below, trace=write is known as a qualifying expression, where trace is a qualifier (others include signal, abbrev, verbose, raw, read, or write). Here, write is the value of the qualifier.

The following command actually shows the system calls to print df output on standard output.

Here are some additional commands about trace qualifier.

8. Trace System Calls Based on a Certain Condition

Let’s look at how to trace system calls relating to a given class of events. This command can be used to trace all system calls involving process management.

Next, to trace all system calls that take a filename as an argument, run this command.

To trace all system calls involving memory mapping, type.

You can trace all network and signals related system calls.

9. Redirect Trace Output to File

To write the trace messages sent to standard error to a file, use the -o option. This means that only the command output is printed on the screen as shown below.

To look through the file, use cat command.

10. Show Some Debugging Output of Strace

To show debugging information for strace tool, use the -d flag.

For additional information, see the strace man page.

Also read these useful related articles:

In conclusion, strace is a remarkable tool for diagnosing cause(s) of program failure: it is a powerful debugging and trouble shooting. It is practically useful to experienced system administrators, programmers and hackers. To share any thoughts concerning this article, use the feedback form below.

If You Appreciate What We Do Here On TecMint, You Should Consider:

TecMint is the fastest growing and most trusted community site for any kind of Linux Articles, Guides and Books on the web. Millions of people visit TecMint! to search or browse the thousands of published articles available FREELY to all.

If you like what you are reading, please consider buying us a coffee ( or 2 ) as a token of appreciation.

We are thankful for your never ending support.

Источник

Dynamic debug¶

Introduction¶

This document describes how to use the dynamic debug (dyndbg) feature.

Dynamic debug is designed to allow you to dynamically enable/disable kernel code to obtain additional kernel information. Currently, if CONFIG_DYNAMIC_DEBUG is set, then all pr_debug() / dev_dbg() and print_hex_dump_debug() / print_hex_dump_bytes() calls can be dynamically enabled per-callsite.

If CONFIG_DYNAMIC_DEBUG is not set, print_hex_dump_debug() is just shortcut for print_hex_dump(KERN_DEBUG) .

For print_hex_dump_debug() / print_hex_dump_bytes() , format string is its prefix_str argument, if it is constant string; or hexdump in case prefix_str is build dynamically.

Dynamic debug has even more useful features:

• Simple query language allows turning on and off debugging statements by matching any combination of 0 or 1 of:
  • source filename
  • function name
  • line number (including ranges of line numbers)
  • module name
  • format string
• Provides a debugfs control file: /dynamic_debug/control which can be read to display the complete list of known debug statements, to help guide you

Controlling dynamic debug Behaviour¶

The behaviour of pr_debug() / dev_dbg() are controlled via writing to a control file in the ‘debugfs’ filesystem. Thus, you must first mount the debugfs filesystem, in order to make use of this feature. Subsequently, we refer to the control file as: /dynamic_debug/control . For example, if you want to enable printing from source file svcsock.c , line 1603 you simply do:

If you make a mistake with the syntax, the write will fail thus:

Viewing Dynamic Debug Behaviour¶

You can view the currently configured behaviour of all the debug statements via:

You can also apply standard Unix text manipulation filters to this data, e.g.:

The third column shows the currently enabled flags for each debug statement callsite (see below for definitions of the flags). The default value, with no flags enabled, is =_ . So you can view all the debug statement callsites with any non-default flags:

Command Language Reference¶

At the lexical level, a command comprises a sequence of words separated by spaces or tabs. So these are all equivalent:

Command submissions are bounded by a write() system call. Multiple commands can be written together, separated by ; or \n :

If your query set is big, you can batch them too:

A another way is to use wildcard. The match rule support * (matches zero or more characters) and ? (matches exactly one character).For example, you can match all usb drivers:

At the syntactical level, a command comprises a sequence of match specifications, followed by a flags change specification:

The match-spec’s are used to choose a subset of the known pr_debug() callsites to which to apply the flags-spec. Think of them as a query with implicit ANDs between each pair. Note that an empty list of match-specs will select all debug statement callsites.

A match specification comprises a keyword, which controls the attribute of the callsite to be compared, and a value to compare against. Possible keywords are::

line-range cannot contain space, e.g. “1-30” is valid range but “1 — 30” is not.

The meanings of each keyword are:

The given string is compared against the function name of each callsite. Example:

The given string is compared against either the full pathname, the src-root relative pathname, or the basename of the source file of each callsite. Examples:

The given string is compared against the module name of each callsite. The module name is the string as seen in lsmod , i.e. without the directory or the .ko suffix and with — changed to _ . Examples:

The given string is searched for in the dynamic debug format string. Note that the string does not need to match the entire format, only some part. Whitespace and other special characters can be escaped using C octal character escape \ooo notation, e.g. the space character is \040 . Alternatively, the string can be enclosed in double quote characters ( » ) or single quote characters ( ‘ ). Examples:

The given line number or range of line numbers is compared against the line number of each pr_debug() callsite. A single line number matches the callsite line number exactly. A range of line numbers matches any callsite between the first and last line number inclusive. An empty first number means the first line in the file, an empty line number means the last number in the file. Examples:

The flags specification comprises a change operation followed by one or more flag characters. The change operation is one of the characters:

For print_hex_dump_debug() and print_hex_dump_bytes() , only p flag have meaning, other flags ignored.

For display, the flags are preceded by = (mnemonic: what the flags are currently equal to).

Note the regexp ^[-+=][flmpt_]+$ matches a flags specification. To clear all flags at once, use =_ or -flmpt .

Debug messages during Boot Process¶

To activate debug messages for core code and built-in modules during the boot process, even before userspace and debugfs exists, use dyndbg=»QUERY» , module.dyndbg=»QUERY» , or ddebug_query=»QUERY» ( ddebug_query is obsoleted by dyndbg , and deprecated). QUERY follows the syntax described above, but must not exceed 1023 characters. Your bootloader may impose lower limits.

These dyndbg params are processed just after the ddebug tables are processed, as part of the arch_initcall. Thus you can enable debug messages in all code run after this arch_initcall via this boot parameter.

On an x86 system for example ACPI enablement is a subsys_initcall and:

will show early Embedded Controller transactions during ACPI setup if your machine (typically a laptop) has an Embedded Controller. PCI (or other devices) initialization also is a hot candidate for using this boot parameter for debugging purposes.

If foo module is not built-in, foo.dyndbg will still be processed at boot time, without effect, but will be reprocessed when module is loaded later. dyndbg_query= and bare dyndbg= are only processed at boot.

Debug Messages at Module Initialization Time¶

When modprobe foo is called, modprobe scans /proc/cmdline for foo.params , strips foo. , and passes them to the kernel along with params given in modprobe args or /etc/modprob.d/*.conf files, in the following order:

parameters given via /etc/modprobe.d/*.conf :

foo.dyndbg as given in boot args, foo. is stripped and passed:

args to modprobe:

These dyndbg queries are applied in order, with last having final say. This allows boot args to override or modify those from /etc/modprobe.d (sensible, since 1 is system wide, 2 is kernel or boot specific), and modprobe args to override both.

In the foo.dyndbg=»QUERY» form, the query must exclude module foo . foo is extracted from the param-name, and applied to each query in QUERY , and only 1 match-spec of each type is allowed.

The dyndbg option is a “fake” module parameter, which means:

• modules do not need to define it explicitly
• every module gets it tacitly, whether they use pr_debug or not
• it doesn’t appear in /sys/module/$module/parameters/ To see it, grep the control file, or inspect /proc/cmdline.

For CONFIG_DYNAMIC_DEBUG kernels, any settings given at boot-time (or enabled by -DDEBUG flag during compilation) can be disabled later via the sysfs interface if the debug messages are no longer needed:

Источник