Tomcat on virtual server: JVM cannot start because 44 GB Ram is not enough ?-Collection of common programming errors

I’m renting a virtual machine, running Ubuntu 9.04. I installed the SUN JDK 6.0 (via apt-get) and Tomcat 6.0.18 (via unzipping).

General problem with Java (partly solved)

At first, it was impossible to run a JVM because of memory problems. Even something simple as java -version failed with

Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

I figured out that even though free reported 44 GB of free memory, I was only allowed to use a small fraction of it, something like 284 MB per allocation. Running java -Xms10m -Xmx256m -version works fine, but I don’t want to change every script that invokes java.

Also I read about the JAVA_OPTS environment variable, which is honored by many apps. I’ve set this to "-Xms10m -Xmx256m"

Problem with Tomcat

The tomcat startup script catalina.sh also uses JAVA_OPTS. But event though, I still get the three error lines mentioned above, readable in logs/catalina.out. I know that JAVA_OPTS is used, because when I put nonsense into that variable, I see the consequences in logs/catalina.out.

It seems like Tomcat also wants to start javac, which needs the options prefixed with -J like -J-Xms10m -J-Xmx256m instead. But I’m not sure if this is related to the problem, because there is no more output than those three lines.

I’m starting Tomcat via bin/startup.sh which in turn calls bin/catalina.sh. I know there is also the possibilty to start it via jsrv, but I cannot do this, because the make / configure process for jsrv also wants to call javac which fails with the well known three line error message mentioned above.

Now I’d be interested in two possible kinds of solution:

• how can I make Java just work on my virtual host, no matter who calls java or javac and no matter how he calls it?
• if this is impossible, how can I adjust Tomcat to just run?

PS: Please note that I only have control over the virtual machine, but no control and only limited knowledge over/about the surrounding physical machine.

PPS: Sorry for the noobish title of this question, but I felt that the original, well written title did not attract enough readers, so I thought this might help 😉 It actually helped, as I got 0 views in the first hour, but 8 views in the hour after the change. Strange world.

Edit: this is the output of ulimit -a:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

1. javac generally spawns a separate process which should fall underneath different process rules.

   -J options are for passing args via a wrapper/launcher to the JVM, I’ve not seen this on Tomcat.

   Find out where the -Xmx is coming from:

   find /your/install/dir/with/tomcat |xargs grep '-Xmx'

2. Try running ulimit -v unlimited before starting tomcat. If this works, you can probably add this to one or both of the startup scripts, or in /etc/profile, ~/.bashrc or some other shell profile script.

3. In lack of alternatives I tried a dirty hack: replacing java and javac by shell scripts which ensure that the proper arguments are passed and allow me to see the exact parameter list. Since this in already an attempt at a very specific solution, and not part of the question, I feld like putting it into an answer.

   I searched for all occurences of java in my filesystem. There were way to many, because I has several JREs and JDKs on my machine. Even after removing all but one version, there were still many of them. I used find / -name "java" |xargs file to find out which is the real one. It told me:

   /var/lib/dpkg/alternatives/java:               ASCII text
   /etc/alternatives/java:                        symbolic link to `/usr/lib/jvm/java-6-sun/jre/bin/java'
   /usr/share/java:                               directory
   /usr/lib/jvm/java-6-sun-1.6.0.16/bin/java:     symbolic link to `../jre/bin/java'
   /usr/lib/jvm/java-6-sun-1.6.0.16/jre/bin/java: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.4.0, not stripped
   /usr/bin/java:                                 symbolic link to `/etc/alternatives/java'
   

   So I knew that /usr/lib/jvm/java-6-sun-1.6.0.16/jre/bin/java was the executable. I renamed it to javaexec and placed a shell script at it’s old location:

   mv /usr/lib/jvm/java-6-sun-1.6.0.16/jre/bin/java /usr/lib/jvm/java-6-sun-1.6.0.16/jre/bin/javaexec
   nano /usr/lib/jvm/java-6-sun-1.6.0.16/jre/bin/java
      (see below for file contents)
   chmod +x #!/bin/bash -e
   

   This ensures that every call to java in the end points to my script, which will call the executable with the right parameters. After that, I did similar terrible things to javac. The contents of my script:

   #!/bin/bash -e
   line=""
   for i in $*
   do
     if [[ "$i" == *Xmx* ]]; then
       echo "Seen -Xmx and removed"
     elif [[ "$i" == *MaxPerm* ]]; then
       echo "Seen -XX:MaxPermSize and removed"
     else
       line="$line $i"
     fi
   done
   exec="/usr/lib/jvm/java-6-sun-1.6.0.16/jre/bin/javaexec -Xms10m -Xmx100m $line"
   echo Now executing $exec
   $exec
   

   It removes each and every parameter from the command line that involves -Xmx or something like -XX:MaxPermSize and puts in its own -Xms and -Xmx limits.

   When I now start Tomcat, it still fails, but finally, the JVM starts up, does something, and then exits with a meaningful message. No reel success, but it leaves me a lot more satisfied, since from there on, I know what to do. While Tomcat still doesn’t run, it has a nice side effect:

   Now I can finally call java or javac on the command line or from a script, and they do not crash, which before was not possible at all! They just run! Hey, I feel like the king of the word, I finally got java to run on a machine with 44GB of Ram without complaing about insufficient memory. This is a great day in the history of computing, indeed!

4. You could also check the java vms that you have installed. Make sure, it’s not the gcj environment that’s giving you problems.

   try

   update-java-alternatives -l
   

   to see if sun-java6 is your default. Drop “-l” to see the other options for this command – e.g. how to change the default.

   If I remember correctly, all java-problems I had on ubuntu had their root in gcj being the default (it might have come in as some dependency prior to sun-java6). If I don’t remember correctly, it’s at least the majority of the problems.

   Last but not least (and only related to your question): You can put a file named “setenv.sh” into tomcat’s bin directory. This file is not there, but will be read on startup if it can be found. You can make all of your changes here in order to refer to them later. The only minor drawback is, that they’ll be read on startup as well as on shutdown.