E.1.3 Debugging mysqld under gdb

On most systems you can also start mysqld from gdb to get more information if mysqld crashes.

With some older gdb versions on Linux you must use run --one-thread if you want to be able to debug mysqld threads. In this case you can only have one thread active at a time. We recommend you to upgrade to gdb 5.1 ASAP as thread debugging works much better with this version!

When running mysqld under gdb, you should disable the stack trace with --skip-stack-trace to be able to catch segfaults within gdb.

In MySQL 4.0.14 and above you should use the --gdb option to mysqld. This will install an interrupt handler for SIGINT (needed to stop mysqld with ^C to set breakpoints) and disable stack tracing and core file handling.

It's very hard to debug MySQL under gdb if you do a lot of new connections the whole time as gdb doesn't free the memory for old threads. You can avoid this problem by starting mysqld with -O thread_cache_size= 'max_connections +1'. In most cases just using -O thread_cache_size=5' will help a lot!

If you want to get a core dump on Linux if mysqld dies with a SIGSEGV signal, you can start mysqld with the --core-file option. This core file can be used to make a backtrace that may help you find out why mysqld died:

shell> gdb mysqld core
gdb>   backtrace full
gdb>   exit

See section A.4.1 What To Do If MySQL Keeps Crashing.

If you are using gdb 4.17.x or above on Linux, you should install a `.gdb' file, with the following information, in your current directory:

set print sevenbit off
handle SIGUSR1 nostop noprint
handle SIGUSR2 nostop noprint
handle SIGWAITING nostop noprint
handle SIGLWP nostop noprint
handle SIGPIPE nostop
handle SIGALRM nostop
handle SIGHUP nostop
handle SIGTERM nostop noprint

If you have problems debugging threads with gdb, you should download gdb 5.x and try this instead. The new gdb version has very improved thread handling!

Here is an example how to debug mysqld:

shell> gdb /usr/local/libexec/mysqld
gdb> run
...
backtrace full # Do this when mysqld crashes

Include the above output in a mail generated with mysqlbug and mail this to mysql@lists.mysql.com.

If mysqld hangs you can try to use some system tools like strace or /usr/proc/bin/pstack to examine where mysqld has hung.

strace /tmp/log libexec/mysqld

If you are using the Perl DBI interface, you can turn on debugging information by using the trace method or by setting the DBI_TRACE environment variable. See section 9.5.2 The DBI Interface.

User Comments

On Solaris you can use instead of strace system
utility called truss. For example, "truss
-t\!ioctl,\!lseek,\!stat,\!fstat,\!fstat64 -e
-vall -xall -a -f ./mysqld"

To debug under a newly started process, chdir() to
the source tree, compile mysql using -g3 CFLAGS
option at the best and run:

# gdb ./sql/mysqld

(gdb) set args --datadir=/data/mysql2
--basedir=/usr/local/mysql-BK20021104
--pid-file=/usr/local/mysql/var/kulan.pidd
--port=3307 --socket=/tmp/mysql2.sock

(gdb) break mysqld.cc:2039

(gdb) break open() <- stop on every open() call

(gdb) run <- run until we reach our breakpoint

(gdb) where <- where are we now in the sources?

(gdb) l <- list source code

(gdb) s <- step

(gdb) n <- next

(gdb) p varname <- print variable contents

(gdb) quit

To debug an already running mysqld process and see
what happens to it if you issue a specific SQL
command from the interactive mysql session, you
have to figure out first to which PID you have to
connect. Issue from you client commands line
"SELECT 3+3" and figure out the PID. (Sinisa says
that you can in principle connect to any mysqld
process and switch context to access another
thread, but NOT on Linux these days yet. Use `info
threads' command at the gdb prompt in such case)

chdir() to unpacked source tree. At the best,
compile next time mysql using CFLAGS="-g3".

# gdb
(gdb) attach 1234
(gdb) detach
(gdb) break sql_parse.cc:981
(gdb) b pthread_mutex_unlock
(gdb) info break
(gdb) attach 12345
(gdb) c
(gdb) s
(gdb) n
(gdb) where
(gdb) l
(gdb) detach
(gdb) quit

Where to set breakpoints as a general start place?
In client in real_connect(). On the server in handle_connections(), but in version above 4.0.8 it got renamed, so use handle_one_connection() for example.

Add your own comment.