Error: Memory leaked by Allocator(ArrowColumnStore) by Loop End-Node
Dear KNIMEler,
I’m getting memory errors from an End Loop node: The message is: LocalNodeExecutionJob : Loop End : 7:1908:1903:0:1423:1689:1683:1216 : Caught “IllegalStateException”: Memory was leaked by query. Memory leaked: (1048576)
Allocator(ArrowColumnStore) 0/1048576/1179648/9223372036854775807 (res/actual/peak/limit)
see attached log for the stack-trace.
This crash happened on a Linux machine with KNIME 4.3.1 using the ‘Columnar Backend’ (lab)
My observation is, that this crash happens (only?) in the context of ‘wide-tables’. I have a WF in place, which loops over each column Ci of a table, computing some statistics and adapting the values of the column Ci accordingly. In other words, I’m transforming a table A (m columns, n rows) into a table B (also having m columns, n rows) - column wise with a Column List-loop and using the values of B for an outlier score (it’s a HBOS implementation with KNIME as a component). This component itself is also executed within a loop - causing the error.
I have a test-setup of 17 outlier data sets - 16 have less than 100 columns, one has 1556! Looping over these data sets evaluating the performance of the implemented outlier detection (HBOS) gets stuck at the wide table!
Any idea?
Best
Erich