oom-killer killing mysql compilation (gentoo)
For days I'm trying to compile mysql on my fresh gentoo installation but it still fails. It's a linode64 using 256mB of swap space plus 256 mB of swap in a file. oom-killer is still killing cc1plus with "all" that memory available. I don't understand well the informations output by dmesg about oom-killer. What mean the numbers about the swap space there?
Can I do something else than upgrading my linode to more RAM to succeed mysql compilation?
Note there is only gkrellmd and sshd daemons running.
Thank you!
$ emerge -av mysql
(...)
if g++ -DEMBEDDED_LIBRARY -DMYSQL_SERVER -DDEFAULT_MYSQL_HOME="\"/usr\"" -DDATADIR="\"/var/lib/mysql\"" -DSHAREDIR="\"/usr/share/mysql\"" -I. -I. -I.. -I../bdb/build_unix -I./../include -I../include -I./.. -I.. -I.. -I../sql -I../regex -O3 -DDBUG_OFF -O2 -march=pentium4 -fomit-frame-pointer -pipe -felide-constructors -fno-exceptions -fno-rtti -fno-implicit-templates -fno-exceptions -fno-rtti -MT sql_select.o -MD -MP -MF ".deps/sql_select.Tpo" \
-c -o sql_select.o `test -f 'sql_select.cc' || echo './'`sql_select.cc; \
then mv -f ".deps/sql_select.Tpo" ".deps/sql_select.Po"; \
else rm -f ".deps/sql_select.Tpo"; exit 1; \
fi
if g++ -DEMBEDDED_LIBRARY -DMYSQL_SERVER -DDEFAULT_MYSQL_HOME="\"/usr\"" -DDATADIR="\"/var/lib/mysql\"" -DSHAREDIR="\"/usr/share/mysql\"" -I. -I. -I.. -I../bdb/build_unix -I./../include -I../include -I./.. -I.. -I.. -I../sql -I../regex -O3 -DDBUG_OFF -O2 -march=pentium4 -fomit-frame-pointer -pipe -felide-constructors -fno-exceptions -fno-rtti -fno-implicit-templates -fno-exceptions -fno-rtti -MT sql_do.o -MD -MP -MF ".deps/sql_do.Tpo" \
-c -o sql_do.o `test -f 'sql_do.cc' || echo './'`sql_do.cc; \
then mv -f ".deps/sql_do.Tpo" ".deps/sql_do.Po"; \
else rm -f ".deps/sql_do.Tpo"; exit 1; \
fi
if g++ -DEMBEDDED_LIBRARY -DMYSQL_SERVER -DDEFAULT_MYSQL_HOME="\"/usr\"" -DDATADIR="\"/var/lib/mysql\"" -DSHAREDIR="\"/usr/share/mysql\"" -I. -I. -I.. -I../bdb/build_unix -I./../include -I../include -I./.. -I.. -I.. -I../sql -I../regex -O3 -DDBUG_OFF -O2 -march=pentium4 -fomit-frame-pointer -pipe -felide-constructors -fno-exceptions -fno-rtti -fno-implicit-templates -fno-exceptions -fno-rtti -MT sql_show.o -MD -MP -MF ".deps/sql_show.Tpo" \
-c -o sql_show.o `test -f 'sql_show.cc' || echo './'`sql_show.cc; \
then mv -f ".deps/sql_show.Tpo" ".deps/sql_show.Po"; \
else rm -f ".deps/sql_show.Tpo"; exit 1; \
fi
if g++ -DEMBEDDED_LIBRARY -DMYSQL_SERVER -DDEFAULT_MYSQL_HOME="\"/usr\"" -DDATADIR="\"/var/lib/mysql\"" -DSHAREDIR="\"/usr/share/mysql\"" -I. -I. -I.. -I../bdb/build_unix -I./../include -I../include -I./.. -I.. -I.. -I../sql -I../regex -O3 -DDBUG_OFF -O2 -march=pentium4 -fomit-frame-pointer -pipe -felide-constructors -fno-exceptions -fno-rtti -fno-implicit-templates -fno-exceptions -fno-rtti -MT sql_string.o -MD -MP -MF ".deps/sql_string.Tpo" \
-c -o sql_string.o `test -f 'sql_string.cc' || echo './'`sql_string.cc; \
then mv -f ".deps/sql_string.Tpo" ".deps/sql_string.Po"; \
else rm -f ".deps/sql_string.Tpo"; exit 1; \
fi
if g++ -DEMBEDDED_LIBRARY -DMYSQL_SERVER -DDEFAULT_MYSQL_HOME="\"/usr\"" -DDATADIR="\"/var/lib/mysql\"" -DSHAREDIR="\"/usr/share/mysql\"" -I. -I. -I.. -I../bdb/build_unix -I./../include -I../include -I./.. -I.. -I.. -I../sql -I../regex -O3 -DDBUG_OFF -O2 -march=pentium4 -fomit-frame-pointer -pipe -felide-constructors -fno-exceptions -fno-rtti -fno-implicit-templates -fno-exceptions -fno-rtti -MT sql_table.o -MD -MP -MF ".deps/sql_table.Tpo" \
-c -o sql_table.o `test -f 'sql_table.cc' || echo './'`sql_table.cc; \
then mv -f ".deps/sql_table.Tpo" ".deps/sql_table.Po"; \
else rm -f ".deps/sql_table.Tpo"; exit 1; \
fi
if g++ -DEMBEDDED_LIBRARY -DMYSQL_SERVER -DDEFAULT_MYSQL_HOME="\"/usr\"" -DDATADIR="\"/var/lib/mysql\"" -DSHAREDIR="\"/usr/share/mysql\"" -I. -I. -I.. -I../bdb/build_unix -I./../include -I../include -I./.. -I.. -I.. -I../sql -I../regex -O3 -DDBUG_OFF -O2 -march=pentium4 -fomit-frame-pointer -pipe -felide-constructors -fno-exceptions -fno-rtti -fno-implicit-templates -fno-exceptions -fno-rtti -MT sql_test.o -MD -MP -MF ".deps/sql_test.Tpo" \
-c -o sql_test.o `test -f 'sql_test.cc' || echo './'`sql_test.cc; \
then mv -f ".deps/sql_test.Tpo" ".deps/sql_test.Po"; \
else rm -f ".deps/sql_test.Tpo"; exit 1; \
fi
if g++ -DEMBEDDED_LIBRARY -DMYSQL_SERVER -DDEFAULT_MYSQL_HOME="\"/usr\"" -DDATADIR="\"/var/lib/mysql\"" -DSHAREDIR="\"/usr/share/mysql\"" -I. -I. -I.. -I../bdb/build_unix -I./../include -I../include -I./.. -I.. -I.. -I../sql -I../regex -O3 -DDBUG_OFF -O2 -march=pentium4 -fomit-frame-pointer -pipe -felide-constructors -fno-exceptions -fno-rtti -fno-implicit-templates -fno-exceptions -fno-rtti -MT sql_udf.o -MD -MP -MF ".deps/sql_udf.Tpo"\
-c -o sql_udf.o `test -f 'sql_udf.cc' || echo './'`sql_udf.cc; \
then mv -f ".deps/sql_udf.Tpo" ".deps/sql_udf.Po"; \
else rm -f ".deps/sql_udf.Tpo"; exit 1; \
fi
if g++ -DEMBEDDED_LIBRARY -DMYSQL_SERVER -DDEFAULT_MYSQL_HOME="\"/usr\"" -DDATADIR="\"/var/lib/mysql\"" -DSHAREDIR="\"/usr/share/mysql\"" -I. -I. -I.. -I../bdb/build_unix -I./../include -I../include -I./.. -I.. -I.. -I../sql -I../regex -O3 -DDBUG_OFF -O2 -march=pentium4 -fomit-frame-pointer -pipe -felide-constructors -fno-exceptions -fno-rtti -fno-implicit-templates -fno-exceptions -fno-rtti -MT sql_update.o -MD -MP -MF ".deps/sql_update.Tpo" \
-c -o sql_update.o `test -f 'sql_update.cc' || echo './'`sql_update.cc; \
then mv -f ".deps/sql_update.Tpo" ".deps/sql_update.Po"; \
else rm -f ".deps/sql_update.Tpo"; exit 1; \
fi
if g++ -DEMBEDDED_LIBRARY -DMYSQL_SERVER -DDEFAULT_MYSQL_HOME="\"/usr\"" -DDATADIR="\"/var/lib/mysql\"" -DSHAREDIR="\"/usr/share/mysql\"" -I. -I. -I.. -I../bdb/build_unix -I./../include -I../include -I./.. -I.. -I.. -I../sql -I../regex -O3 -DDBUG_OFF -O2 -march=pentium4 -fomit-frame-pointer -pipe -felide-constructors -fno-exceptions -fno-rtti -fno-implicit-templates -fno-exceptions -fno-rtti -MT sql_yacc.o -MD -MP -MF ".deps/sql_yacc.Tpo" \
-c -o sql_yacc.o `test -f 'sql_yacc.cc' || echo './'`sql_yacc.cc; \
then mv -f ".deps/sql_yacc.Tpo" ".deps/sql_yacc.Po"; \
else rm -f ".deps/sql_yacc.Tpo"; exit 1; \
fi
g++: Internal error: Terminated (program cc1plus)
Please submit a full bug report.
See <url:http: bugs.gentoo.org=""> for instructions.
make[3]: *** [sql_yacc.o] Error 1
make[3]: Leaving directory `/var/tmp/portage/mysql-4.0.22/work/mysql-4.0.22/libmysqld'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/var/tmp/portage/mysql-4.0.22/work/mysql-4.0.22/libmysqld'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/var/tmp/portage/mysql-4.0.22/work/mysql-4.0.22'
make: *** [all] Error 2
!!! ERROR: dev-db/mysql-4.0.22 failed.
!!! Function src_compile, Line 160, Exitcode 2
!!! compile problem
!!! If you need support, post the topmost build error, NOT this status message.</url:http:>
$ dmesg
(...)
Free pages: 256kB (0kB HighMem)
Active:12141 inactive:407 dirty:0 writeback:0 unstable:0 free:64 slab:1842 mapped:11757 pagetables:109
DMA free:256kB min:256kB low:512kB high:768kB active:48564kB inactive:1628kB present:65536kB
protections[]: 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 2*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 256kB
Normal: empty
HighMem: empty
Swap cache: add 10502, delete 10360, find 2921/3493, race 0+0
Out of Memory: Killed process 18555 (cc1plus).
oom-killer: gfp_mask=0x1d2
DMA per-cpu:
cpu 0 hot: low 8, high 24, batch 4
cpu 0 cold: low 0, high 8, batch 4
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages: 248kB (0kB HighMem)
Active:12298 inactive:225 dirty:0 writeback:0 unstable:0 free:62 slab:1846 mapped:11732 pagetables:126
DMA free:248kB min:256kB low:512kB high:768kB active:49192kB inactive:900kB present:65536kB
protections[]: 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
DMA: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 248kB
Normal: empty
HighMem: empty
Swap cache: add 16477, delete 16414, find 5536/6541, race 0+0
Out of Memory: Killed process 19055 (sshd).
oom-killer: gfp_mask=0xd2
DMA per-cpu:
cpu 0 hot: low 8, high 24, batch 4
cpu 0 cold: low 0, high 8, batch 4
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages: 744kB (0kB HighMem)
Active:12465 inactive:0 dirty:0 writeback:0 unstable:0 free:186 slab:1783 mapped:11705 pagetables:113
DMA free:744kB min:256kB low:512kB high:768kB active:49860kB inactive:0kB present:65536kB
protections[]: 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
DMA: 92*4kB 13*8kB 3*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 744kB
Normal: empty
HighMem: empty
Swap cache: add 28538, delete 25596, find 7949/10099, race 0+0
Out of Memory: Killed process 11811 (cc1plus).
oom-killer: gfp_mask=0xd2
DMA per-cpu:
cpu 0 hot: low 8, high 24, batch 4
cpu 0 cold: low 0, high 8, batch 4
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages: 584kB (0kB HighMem)
Active:11833 inactive:633 dirty:0 writeback:0 unstable:0 free:146 slab:1821 mapped:11706 pagetables:115
DMA free:584kB min:256kB low:512kB high:768kB active:47332kB inactive:2532kB present:65536kB
protections[]: 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
DMA: 34*4kB 14*8kB 7*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 584kB
Normal: empty
HighMem: empty
Swap cache: add 44798, delete 41944, find 12964/16526, race 0+0
Out of Memory: Killed process 2434 (cc1plus).
9 Replies
Post the output of
cat /proc/io_status
and
vmstat -s
so we can see if anything is broken.
nyat root # cat /proc/iostatus
iocount=606999 iorate=0 iotokens=399995 tokenrefill=512 tokenmax=400000
nyat root # vmstat -s
59512 total memory
58376 used memory
11132 active memory
27820 inactive memory
1136 free memory
27652 buffer memory
8588 swap cache
528572 total swap
2600 used swap
525972 free swap
254995 non-nice user cpu ticks
3412 nice user cpu ticks
1225387 system cpu ticks
15639522 idle cpu ticks
113930 IO-wait cpu ticks
28 IRQ cpu ticks
23613 softirq cpu ticks
1212233 pages paged in
1215856 pages paged out
24937 pages swapped in
23496 pages swapped out
18405981 interrupts
3062915 CPU context switches
1104972404 boot time
231341 forks
````
some hours after the last mysql compilation attempt (without reboot). There is nothing strange to me there..
@jbpros:
some hours after the last mysql compilation attempt (without reboot). There is nothing strange to me there..
Your io_status and vmstat do indeed look normal.
@jbpros:
I've been using the -march=pentium4 flag in /etc/make.conf. I read old stuffs about this flag being problematic. Any comment?
My Linode is a Gentoo stage one install with
CHOST="i686-pc-linux-gnu"
CFLAGS="-march=pentium4 -O2 -pipe -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}"
MAKEOPTS="-j2"
and it has compiled MySQL OK
nyat root # emerge info
Portage 2.0.51-r3 (default-linux/x86/2004.3, gcc-3.3.4, glibc-2.3.4.20040808-r1, 2.6.9-linode9 i686)
System uname: 2.6.9-linode9 i686 UML
Gentoo Base System version 1.4.16
ccache version 2.3 [enabled]
Autoconf: sys-devel/autoconf-2.59-r5
Automake: sys-devel/automake-1.8.5-r1
Binutils: sys-devel/binutils-2.15.90.0.1.1-r3
Headers: sys-kernel/linux26-headers-2.6.8.1-r1
Libtools: sys-devel/libtool-1.5.2-r7
ACCEPTKEYWORDS="x86"
AUTOCLEAN="yes"
CFLAGS="-O2 -march=pentium4 -fomit-frame-pointer -pipe"
CHOST="i686-pc-linux-gnu"
COMPILER=""
CONFIGPROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control"
CONFIGPROTECTMASK="/etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-O2 -march=pentium4 -fomit-frame-pointer -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoaddcvs autoconfig ccache distlocks sandbox sfperms"
GENTOOMIRRORS="http://mirror.datapipe.net/gentoo http://prometheus.cs.wmich.edu/gentoo http://mirror.datapipe.net/gentoo"
MAKEOPTS="-j1"
PKGDIR="/usr/portage/packages"
PORTAGETMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.us.gentoo.org/gentoo-portage"
USE="x86 acl apache2 apm arts avi berkdb bitmap-fonts cgi crypt cups dba encode foomaticdb fortran freetype gd gdbm gif imap ipv6 jabber jpeg libwww mad maildir md5sum mikmod mime motif mpeg msn mysql ncurses nls nptlonly oggvorbis oscar oss pam pdflib perl php png ppds python quicktime readline sdl spell ssl svga tcpd truetype xml2 xmms yahoo zlib"
nyat root # emerge -av mysql;
These are the packages that I would merge, in order:
Calculating dependencies …done!
[ebuild N ] dev-db/mysql-4.0.22 +berkdb -debug -innodb +perl +readline (-selinux) +ssl -static +tcpd 0 kB
[ebuild N ] dev-perl/DBD-mysql-2.1027 0 kB
Total size of downloads: 0 kB
nyat root # cat /etc/make.conf
These settings were set by the catalyst build script that automatically built this stage
--- Host-specific flags & Use flags ---
CFLAGS="-O2 -march=pentium4 -fomit-frame-pointer -pipe"
CHOST="i686-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"
USE="apache2 cgi dba freetype gd -gnome -gpm -gtk -gtk2 imap jabber -kde maildir md5sum mime msn mysql nptlonly oggvorbis oscar php ppds -qt yahoo -opengl -xpm -xv -X"
--- Compilation flags ---
MAKEOPTS="-j1"
FEATURES="ccache"
PORTAGE_NICENESS="0"
--- Directories ---
PORTDIR="/usr/portage"
DISTDIR="${PORTDIR}/distfiles"
PORTDIROVERLAY="/usr/local/portage"
DISTCCDIR="/var/tmp/portage/.distcc"
--- Mirrors ---
GENTOO_MIRRORS="http://mirror.datapipe.net/gentoo http://prometheus.cs.wmich.edu/gentoo http://mirror.datapipe.net/gentoo"
SYNC="rsync://rsync.us.gentoo.org/gentoo-portage"
````
Note: I'm now trying again the compilation without ccache.
I just noticed the "nptlonly" flag being there. Does it have some effect when nptl is off? I think not but maybe I'm wrong?
Do you notice something else wrong?
Thank you for your kind help.
> If you get errors when compiling `sql_yacc.cc', such as the ones shown here, you have probably run out of memory or swap space:
Internal compiler error: program cc1plus got fatal signal 11
or
Out of virtual memory
or
Virtual memory exhausted
The problem is that gcc requires huge amounts of memory to compile `sql_yacc.cc' with inline functions. Try running configure with the –with-low-memory option:
shell> ./configure --with-low-memory
This option causes -fno-inline to be added to the compile line if you are using gcc and -O0 if you are using something else. You should try the --with-low-memory option even if you have so much memory and swap space that you think you can't possibly have run out. This problem has been observed to occur even on systems with generous hardware configurations, and the --with-low-memory option usually fixes it.
I thus added -fno-inline to my CFLAGS. It's compiling now, results in a while.
Anyway I'm still not convinced that 512mb of swap + 64mb of RAM would have caused OOM situations when compiling mysql…
Thanks for your help.
