PermGen error in Java, while memory usage seems low-Collection of common programming errors

Java application dies from time to time with PermGen space error, but when I look at the memory usage it seems low from what I can tell.

It is a Tomcat application, plus there is SOLR server running (under the same tomcat).

JVM params in catalina.sh:

-Xms1024m 
-Xmx2048m 
-XX:+HeapDumpOnOutOfMemoryError 
-XX:HeapDumpPath=/path/to/logs

Error in catalina.out:

java.lang.OutOfMemoryError: PermGen space
Dumping heap to /path/to/logs/java_pid22335.hprof ...
Heap dump file created [107041478 bytes in 1.823 secs]
Exception in thread "pool-5-thread-1" java.lang.OutOfMemoryError: PermGen space

Now the first strange thing is that memory dump is only 100Mb, while heap limit is 2048Mb. When I used to have “proper” out of memory errors, dump file was close in size to the heap limit.

Second strange thing is memory usage shown by jmap -heap 22335 seems pretty normal (when this command was run the java app was still down):

Attaching to process ID 22335, please wait...   
Debugger attached successfully.
Server compiler detected.
JVM version is 11.3-b02

using thread-local object allocation.
Parallel GC with 16 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 2147483648 (2048.0MB)
   NewSize          = 2686976 (2.5625MB)
   MaxNewSize       = -65536 (-0.0625MB)
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 21757952 (20.75MB)
   MaxPermSize      = 88080384 (84.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 566689792 (540.4375MB)
   used     = 13467648 (12.84375MB)
   free     = 553222144 (527.59375MB)
   2.3765467792297907% used
From Space:
   capacity = 327680 (0.3125MB)
   used     = 0 (0.0MB)
   free     = 327680 (0.3125MB)
   0.0% used
To Space:
   capacity = 327680 (0.3125MB)
   used     = 0 (0.0MB)
   free     = 327680 (0.3125MB)
   0.0% used
PS Old Generation
   capacity = 1431699456 (1365.375MB)
   used     = 86216248 (82.22222137451172MB)
   free     = 1345483208 (1283.1527786254883MB)
   6.021951579200712% used
PS Perm Generation
   capacity = 88080384 (84.0MB)
   used     = 88080080 (83.99971008300781MB)
   free     = 304 (2.899169921875E-4MB)
   99.99965486072358% used

I looked at the memory dump file, nothing unusual there. Tried increasing heap limit, no difference.

Any ideas what could it be and why the dump file is so small, so is the memory usage while the app continues throwing out of memory errors?

1. PS Perm Generation
      capacity = 88080384 (84.0MB)
      used     = 88080080 (83.99971008300781MB)
      free     = 304 (2.899169921875E-4MB)
      99.99965486072358% used
   

   There’s your culprit. PermGen is used for class definitions, and you seem to have a lot of them!

   Try cranking up your PermGen size by adding this to your Tomcat launch options:

   -XX:MaxPermSize=256M
   

   …but if your application is smashing into your current limit, this change will only give you some extra time between crashes.

   Monitor how it behaves and how quickly the usage grows, but be ready to do some digging in your application code for problem class definitions.

2. Max PermGen size is set by a separate command line switch, try -XX:MaxPermSize=128M, that should be enough for PermGen.