Tomcat dying silently on regular basis-Collection of common programming errors
My tomcat (6.0.32, Java Sun 1.6.0_22-b04 on Ubuntu 10.04) keeps crashing multiple times daily without any specific output in catalina.out. This usually happens on high load (see top output). Update: The pid-file is properly removed when this happens.
Update 2: No CATALINA_OPTS set, _JAVA_OPTS are:
export _JAVA_OPTIONS="-Xms128m -Xmx1024m -XX:MaxPermSize=512m \
-XX:MinHeapFreeRatio=20 \
-XX:MaxHeapFreeRatio=40 \
-XX:NewSize=10m \
-XX:MaxNewSize=10m \
-XX:SurvivorRatio=6 \
-XX:TargetSurvivorRatio=80 \
-XX:+CMSClassUnloadingEnabled \
-Djava.awt.headless=true \
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=37331 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=true \
-Djava.rmi.server.hostname=(myhostname) \
-Dcom.sun.management.jmxremote.password.file=/etc/java-6-sun/management/jmxremote.password \
-Dcom.sun.management.jmxremote.access.file=/etc/java-6-sun/management/jmxremote.access"
Top:
top - 12:40:03 up 9 days, 12:15, 3 users, load average: 30.00, 22.39, 21.91
Tasks: 89 total, 4 running, 85 sleeping, 0 stopped, 0 zombie
Cpu(s): 53.2%us, 9.7%sy, 0.0%ni, 34.7%id, 1.5%wa, 0.0%hi, 0.8%si, 0.0%st
Mem: 4194304k total, 3311304k used, 883000k free, 0k buffers
Swap: 4194304k total, 0k used, 4194304k free, 0k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25850 tomcat6 20 0 1981m 1.2g 11m S 161 29.6 11:41.56 java
12632 mysql 20 0 393m 97m 4452 S 141 2.4 1690:05 mysqld
14932 nobody 20 0 253m 44m 9152 R 56 1.1 3:26.57 php-cgi
7011 nobody 20 0 241m 31m 9124 S 30 0.8 1:35.96 php-cgi
10093 nobody 20 0 228m 18m 8520 S 25 0.5 2:29.97 php-cgi
27071 nobody 20 0 237m 28m 8640 S 11 0.7 3:13.72 php-cgi
3306 nobody 20 0 227m 16m 6736 R 7 0.4 2:29.83 php-cgi
7756 nobody 20 0 261m 58m 15m R 5 1.4 2:22.33 php-cgi
7129 www-data 20 0 3646m 7228 1896 S 2 0.2 0:36.65 nginx
2657 nobody 20 0 228m 18m 8540 S 1 0.5 1:59.51 php-cgi
7131 www-data 20 0 3645m 6464 1960 S 1 0.2 0:34.13 nginx
7140 www-data 20 0 3652m 12m 1896 S 1 0.3 0:35.80 nginx
619 nobody 20 0 231m 29m 15m S 0 0.7 2:33.46 php-cgi
16552 nobody 20 0 250m 41m 8784 S 0 1.0 2:48.12 php-cgi
17134 nobody 20 0 239m 37m 16m S 0 0.9 2:32.86 php-cgi
21004 nobody 20 0 243m 34m 8700 S 0 0.8 1:19.85 php-cgi
26105 root 20 0 19220 1392 1060 R 0 0.0 0:00.82 top
32430 nobody 20 0 256m 47m 9196 S 0 1.2 2:19.01 php-cgi
314 nobody 20 0 256m 47m 8804 S 0 1.1 1:46.00 php-cgi
2111 nobody 20 0 253m 44m 9196 S 0 1.1 3:01.14 php-cgi
2142 root 20 0 26452 2564 868 S 0 0.1 0:00.56 screen
2144 root 20 0 19484 2012 1368 S 0 0.0 0:00.00 bash
2333 nobody 20 0 249m 41m 9160 S 0 1.0 1:10.33 php-cgi
2552 root 20 0 19484 2260 1620 S 0 0.1 0:00.01 bash
2587 nobody 20 0 258m 49m 9192 S 0 1.2 2:04.50 php-cgi
2684 root 20 0 4092 652 540 S 0 0.0 0:00.00 xvfb-run
2696 root 20 0 60720 13m 2352 S 0 0.3 0:09.12 Xvfb
2759 root 20 0 617m 12m 4676 S 0 0.3 0:00.66 node
3514 nobody 20 0 270m 61m 9216 S 0 1.5 3:13.69 php-cgi
5270 root 20 0 25164 1324 1036 S 0 0.0 0:00.01 screen
5402 nobody 20 0 227m 16m 8032 S 0 0.4 1:33.61 php-cgi
5765 root 20 0 81180 3820 3028 S 0 0.1 0:00.31 sshd
5798 nobody 20 0 242m 32m 9124 S 0 0.8 1:52.08 php-cgi
5856 root 20 0 19496 2292 1636 S 0 0.1 0:00.03 bash
6442 root 20 0 62332 20m 1960 S 0 0.5 0:30.58 mrtg
7082 root 20 0 88992 1916 1636 S 0 0.0 0:00.00 PassengerWatchd
I can’t find any concrete reason for it, no Exceptions or messages of a shutdown in catalina.out (and no other logs in tomcat’s log dir). I can start up the service and it will run for a few days or just minutes before dying again.
Is there somewhere else i could look for output? Could the kernel start killing threads due to a lack of ressources and by that bring the VM down?