MySQL fails to start

About

MySQL process fails to start.

Looks like this:

 [root@hostname ~]# service mysqld restart
 Stopping mysqld:                                           [  OK  ]
 MySQL Daemon failed to start.
 Starting mysqld:                                           [FAILED]

Step 0 (reboot)

Restart the server:

reboot

Check after restart if mysql is working:

 service mysqld status

Step 1 (check free HDD space)

Check if there is left free space on disk. If disk is full, remove unnecessary files and start MySQL service again.

Example which shows full (100%) /dev/md2:

[root@hostname]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/md2        204G  204G   16G  100% /
tmpfs            16G     0   16G   0% /dev/shm
/dev/md1        496M   59M  412M  13% /boot

Step 2 (check log)

Check if there is any information in log file /var/log/mysqld.log

If there is no NEW information in this log file, do:

/etc/init.d/mysqld restart

and recheck. If still there is no NEW information in this log file, set such permissions with chmod:

• /var/lib/mysql - 755
• files within /var/lib/mysql - 660
• and its subdirectories - 700

Check if you set permissions correctly:

find /var/lib/mysql -printf '%m %p\n'

Do again:

/etc/init.d/mysqld restart

MySQL can start. If not - check log file again, some new info should appear in it (usually error 13)

Step 3 (error 13)

If you see such error in log:

InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
InnoDB: File name ./ib_logfile0
InnoDB: File operation call: 'open'.
InnoDB: Cannot continue operation.

Then execute:

chown -R mysql:mysql  /var/lib/mysql

And start MySQL again:

/etc/init.d/mysqld restart

If MySQL still fails to start do:

chmod 777 /var/lib/mysql/*
/etc/init.d/mysqld restart

If MySQL still fails to start, check if there are any information in log.

Step 4 (error 13 /tmp)

If you get this error:

Can't create/write to file '/tmp/ib0iEPXa' (Errcode: 13)
InnoDB: Error: unable to create temporary file; errno: 13
[ERROR] Plugin 'InnoDB' init function returned error.
[ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
[ERROR] Unknown/unsupported storage engine: InnoDB
[ERROR] Aborting

Do:

mkdir -p /tmp
chmod 1777 /tmp

Restart mysql, if fails - check log.

Step 5 (messed up root permissions)

ERROR 1044 (42000): Access denied for user 'root'@'localhost'

Run:

mysqld_safe --skip-grant-table

And give all privileges to the root.

Step 6 (crashing)

Log file looks like this:

/usr/libexec/mysqld() [0x787c3d]
/usr/libexec/mysqld(recv_recover_page+0x34f) [0x7896ff]
/usr/libexec/mysqld(buf_page_io_complete+0x548) [0x74aab8]
/usr/libexec/mysqld(fil_aio_wait+0xfa) [0x762c6a]
/usr/libexec/mysqld() [0x7c8ce0]
/lib64/libpthread.so.0() [0x3386807aa1]
/lib64/libc.so.6(clone+0x6d) [0x33864e8bcd]

Make DB file backups and reinstall or proceed here: https://dev.mysql.com/doc/refman/8.0/en/crashing.html

Step 7

Try to remove ib_logfile0 and ib_logfile1, (with backups):

mv -fr /var/lib/mysql/ib_logfile* /home/

And start MySQL again:

/etc/init.d/mysqld restart

If MySQL still fails to start, check if there any information in log (/var/log/mysqld.log).

Step 8

Run this command to check more information when MySQL starts:

sh -x /etc/init.d/mysqld start

Example:

[root@CentOS-69-64-minimal ~]# sh -x /etc/init.d/mysqld start
+ . /etc/rc.d/init.d/functions
++ TEXTDOMAIN=initscripts
++ umask 022
++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
++ export PATH
++ '[' -z  ']'
++ COLUMNS=80
++ '[' -z  ']'
+++ /sbin/consoletype
++ CONSOLETYPE=pty
++ '[' -f /etc/sysconfig/i18n -a -z  -a -z  ']'
++ . /etc/profile.d/lang.sh
++ unset LANGSH_SOURCED
++ '[' -z  ']'
++ '[' -f /etc/sysconfig/init ']'
++ . /etc/sysconfig/init
+++ BOOTUP=color
+++ RES_COL=60
+++ MOVE_TO_COL='echo -en \033[60G'
+++ SETCOLOR_SUCCESS='echo -en \033[0;32m'
+++ SETCOLOR_FAILURE='echo -en \033[0;31m'
+++ SETCOLOR_WARNING='echo -en \033[0;33m'
+++ SETCOLOR_NORMAL='echo -en \033[0;39m'
+++ PROMPT=yes
+++ AUTOSWAP=no
+++ ACTIVE_CONSOLES='/dev/tty[1-6]'
+++ SINGLE=/sbin/sushell
++ '[' pty = serial ']'
++ __sed_discard_ignored_files='/\(~\|\.bak\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d'
+++ cat /proc/cmdline
++ strstr 'ro root=/dev/md/2 rd_NO_LUKS rd_NO_DM nomodeset   consoleblank=0 crashkernel=131M@48M SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=de' rc.debug
++ '[' 'ro root=/dev/md/2 rd_NO_LUKS rd_NO_DM nomodeset   consoleblank=0 crashkernel=131M@48M SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=de' = 'ro root=/dev/md/2 rd_NO_LUKS rd_NO_DM nomodeset   consoleblank=0 crashkernel=131M@48M SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=de' ']'
++ return 1
+ . /etc/sysconfig/network
++ NETWORKING=yes
++ HOSTNAME=CentOS-69-64-minimal
+ exec=/usr/bin/mysqld_safe
+ prog=mysqld
+ STARTTIMEOUT=120
+ STOPTIMEOUT=60
+ MYOPTIONS=
+ '[' -e /etc/sysconfig/mysqld ']'
+ . /etc/sysconfig/mysqld
++ STARTTIMEOUT=120
++ STOPTIMEOUT=60
++ MYOPTIONS=
+ lockfile=/var/lock/subsys/mysqld
+ get_mysql_option mysqld datadir /var/lib/mysql
++ /usr/bin/my_print_defaults mysqld
++ sed -n 's/^--datadir=//p'
++ tail -n 1
+ result=/var/lib/mysql
+ '[' -z /var/lib/mysql ']'
+ datadir=/var/lib/mysql
+ get_mysql_option mysqld socket /var/lib/mysql/mysql.sock
++ /usr/bin/my_print_defaults mysqld
++ sed -n 's/^--socket=//p'
++ tail -n 1
+ result=/var/lib/mysql/mysql.sock
+ '[' -z /var/lib/mysql/mysql.sock ']'
+ socketfile=/var/lib/mysql/mysql.sock
+ get_mysql_option mysqld_safe log-error /var/log/mysqld.log
++ /usr/bin/my_print_defaults mysqld_safe
++ sed -n 's/^--log-error=//p'
++ tail -n 1
+ result=/var/log/mysqld.log
+ '[' -z /var/log/mysqld.log ']'
+ errlogfile=/var/log/mysqld.log
+ get_mysql_option mysqld_safe pid-file /var/run/mysqld/mysqld.pid
++ /usr/bin/my_print_defaults mysqld_safe
++ sed -n 's/^--pid-file=//p'
++ tail -n 1
+ result=/var/run/mysqld/mysqld.pid
+ '[' -z /var/run/mysqld/mysqld.pid ']'
+ mypidfile=/var/run/mysqld/mysqld.pid
+ case "$1" in
+ start
+ '[' -x /usr/bin/mysqld_safe ']'
++ /usr/bin/mysqladmin --socket=/var/lib/mysql/mysql.sock --user=UNKNOWN_MYSQL_USER ping
+ RESPONSE='/usr/bin/mysqladmin: connect to server at '\localhost'\ failed
error: '\Can'\t connect to local MySQL server through socket '\/var/lib/mysql/mysql.sock'\ (2)'\
Check that mysqld is running and that the socket: '\/var/lib/mysql/mysql.sock'\ exists!'
+ '[' 1 = 0 ']'
+ echo '/usr/bin/mysqladmin: connect to server at '\localhost'\ failed
error: '\Can'\t connect to local MySQL server through socket '\/var/lib/mysql/mysql.sock'\ (2)'\
Check that mysqld is running and that the socket: '\/var/lib/mysql/mysql.sock'\ exists!'
+ grep -q 'Access denied for user'
+ touch /var/log/mysqld.log
+ chown mysql:mysql /var/log/mysqld.log
+ chmod 0640 /var/log/mysqld.log
+ '[' -x /sbin/restorecon ']'
+ /sbin/restorecon /var/log/mysqld.log
+ '[' '!' -d /var/lib/mysql/mysql ']'
+ chown mysql:mysql /var/lib/mysql
+ chmod 0755 /var/lib/mysql
+ '['  = sos ']'
+ safe_pid=18744
+ ret=0
+ TIMEOUT=120
+ '[' 120 -gt 0 ']'
+ /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
++ /usr/bin/mysqladmin --socket=/var/lib/mysql/mysql.sock --user=UNKNOWN_MYSQL_USER ping
+ RESPONSE='/usr/bin/mysqladmin: connect to server at '\localhost'\ failed
error: '\Can'\t connect to local MySQL server through socket '\/var/lib/mysql/mysql.sock'\ (2)'\
Check that mysqld is running and that the socket: '\/var/lib/mysql/mysql.sock'\ exists!'
+ mret=1
+ '[' 1 -eq 0 ']'
+ '[' 1 -ne 1 -a 1 -ne 11 ']'
+ echo '/usr/bin/mysqladmin: connect to server at '\localhost'\ failed
error: '\Can'\t connect to local MySQL server through socket '\/var/lib/mysql/mysql.sock'\ (2)'\
Check that mysqld is running and that the socket: '\/var/lib/mysql/mysql.sock'\ exists!'
+ grep -q 'Access denied for user'
+ /bin/kill -0 18744
+ sleep 1
+ let TIMEOUT=120-1
+ '[' 119 -gt 0 ']'
++ /usr/bin/mysqladmin --socket=/var/lib/mysql/mysql.sock --user=UNKNOWN_MYSQL_USER ping
+ RESPONSE='/usr/bin/mysqladmin: connect to server at '\localhost'\ failed
error: '\Can'\t connect to local MySQL server through socket '\/var/lib/mysql/mysql.sock'\ (2)'\
Check that mysqld is running and that the socket: '\/var/lib/mysql/mysql.sock'\ exists!'
+ mret=1
+ '[' 1 -eq 0 ']'
+ '[' 1 -ne 1 -a 1 -ne 11 ']'
+ echo '/usr/bin/mysqladmin: connect to server at '\localhost'\ failed
error: '\Can'\t connect to local MySQL server through socket '\/var/lib/mysql/mysql.sock'\ (2)'\
Check that mysqld is running and that the socket: '\/var/lib/mysql/mysql.sock'\ exists!'
+ grep -q 'Access denied for user'
+ /bin/kill -0 18744
+ echo 'MySQL Daemon failed to start.'
MySQL Daemon failed to start.
+ ret=1
+ break
+ '[' 119 -eq 0 ']'
+ '[' 1 -eq 0 ']'
+ action 'Starting mysqld: ' /bin/false
+ local STRING rc
+ STRING='Starting mysqld: '
+ echo -n 'Starting mysqld:  '
Starting mysqld:  + shift
+ /bin/false
+ failure 'Starting mysqld: '
+ local rc=1
+ '[' color '!=' verbose -a -z  ']'
+ echo_failure
+ '[' color = color ']'
+ echo -en '\033[60G'
                                                           + echo -n '['
[+ '[' color = color ']'
+ echo -en '\033[0;31m'
+ echo -n FAILED
FAILED+ '[' color = color ']'
+ echo -en '\033[0;39m'
+ echo -n ']'
]+ echo -ne '\r'
+ return 1
+ '[' -x /bin/plymouth ']'
+ /bin/plymouth --details
+ return 1
+ rc=1
+ echo
+ return 1
+ return 1
+ exit 1
[root@CentOS-69-64-minimal ~]#

See also

• https://dev.mysql.com/doc/refman/5.7/en/starting-server-troubleshooting.html
• https://dev.mysql.com/doc/refman/5.5/en/error-creating-innodb.html