How to get crash info from core file with GDB

Install gdb:

yum install -y gdb

Go to /tmp folder:

cd /tmp

Note. On Asterisk 15 systems core files are located in /var/lib/asterisk.

Check for core files which are generated when Asterisk crashes:

ls

You should see something like this:

[root@billing tmp]# ls
hsperfdata_root  
mor_ast_h323.log  
mor_crash.log
core.billing-2009-11-13T13:58:06+0600
core.billing-2009-11-16T14:53:09+0600 
filegt1h7z                                     
mc-root          
mor_ast_reg.log   
mor_debug.txt

Files marked in bold are core files which we are interested in.

Select one core file, let's say second one and issue:

gdb asterisk core.billing-2009-11-16T14:53:09+0600

you should see a lot of output which is not interesting for us, when it stops issue:

bt

and you will get output which should help to determine the cause of the crash.

As example this output leads us to think the crash was due to h323 module:

(gdb) bt
#0  0x01909008 in PThread::PXBlockOnIO () from /usr/local/lib/libpt_linux_x86_r.so.1.10.0
#1  0x018fcef9 in PChannel::PXSetIOBlock () from /usr/local/lib/libpt_linux_x86_r.so.1.10.0
#2  0x018f8c7b in PSocket::os_sendto () from /usr/local/lib/libpt_linux_x86_r.so.1.10.0
#3  0x0190f6af in PTCPSocket::Write () from /usr/local/lib/libpt_linux_x86_r.so.1.10.0
#4  0x018fdc94 in PIndirectChannel::Write () from /usr/local/lib/libpt_linux_x86_r.so.1.10.0
#5  0x00e3b84c in H323TransportTCP::WritePDU () from /usr/local/lib/libh323_linux_x86_r.so.1.18.0
#6  0x00e1b1fb in H323SignalPDU::Write () from /usr/local/lib/libh323_linux_x86_r.so.1.18.0
#7  0x00df5537 in H323Connection::WriteSignalPDU () from /usr/local/lib/libh323_linux_x86_r.so.1.18.0
#8  0x00df7502 in H323Connection::SetCallEndReason () from /usr/local/lib/libh323_linux_x86_r.so.1.18.0
#9  0x014366bd in h323_clear_call () from /usr/lib/asterisk/modules/chan_h323.so
#10 0x0142deb9 in oh323_hangup (c=0xa86fdd8) at chan_h323.c:736
#11 0x08083877 in ast_hangup (chan=0xa86fdd8) at channel.c:1479
#12 0x03fdff8e in dial_exec_full (chan=0xb7de29d8, data=<value optimized out>, peerflags=0x40f1194, continue_exec=0x0) at app_dial.c:323
#13 0x03fe40a2 in dial_exec (chan=0xb7de29d8, data=0x40f1cd8) at app_dial.c:1760
#14 0x080bfabe in pbx_exec (c=0xb7de29d8, app=0xb7d02588, data=0x40f1cd8) at pbx.c:532
#15 0x046d18d2 in ?? () from /usr/lib/asterisk/modules/app_mor.so
#16 0xb7de29d8 in ?? ()
#17 0xb7d02588 in ?? ()
#18 0x040f1cd8 in ?? ()
#19 0x00000001 in ?? ()
#20 0x0000013f in ?? ()
#21 0x00000000 in ?? ()

Get more details

Sometimes gdb does not provide enough details for troubleshooting. Then it is necessary to recompile Asterisk with some options enabled.

cd /usr/src/asterisk
make clean
make menuconfig

Press x

make
make install

cd /usr/src/asterisk-addons
make clean
make install

Then:

1. Recompile asterisk-addons
2. Recompile MOR Core
3. Restart Asterisk
4. Wait for crash
5. Analyze core file with gdb asterisk ....