Xmx parameter doesn't prevent JVM using RAM beyond it's limit.
In reference to this topic I have exactely the same problem under linux but not on Windows : Memory problems causing server to shut down KNIME (it's driving me crazy) - #27 by christian.birkhold
Problem : tests are done on a windows laptop (32G ram it works Xmx 20G) but production environment are under Linux (48G RAM Xmx 40 or 20 or 30 whatever Memory grows to 48 freezing shutdown under 5.3 => 5.8 Knime version). So I ask the question again, why Xmx parameter doesn’t work on linux ?