RAM Error with H2O AutoML Extension in KNIME During Regression Task

Description:

I’m experiencing a RAM-related issue when using the H2O AutoML extension in KNIME during a regression task. The error occurs after modifying my dataset. Below are the details:

Current Workflow:

  • I’m reading data from two different CSV files, performing operations on them, and joining them. One intensive operation involves adding a feature based on previous rows.

  • The final dataset contains approximately 60,000 rows and 300 columns (previously 60,000 rows and 30 columns).

  • My KNIME installation is set to use up to 28 GB of RAM. This worked fine for previous regression runs, but the issue arose after I expanded the dataset (by adding data from up to 295 previous rows, whereas I previously only added data from 50 rows).

Problem:

  • After modifying the dataset, the learner node crashes with a Java Heap Memory error.

  • When using a column filter to reduce the dataset size before the H2O table creation, the learner node no longer crashes, but gets stuck at 2%.

  • I tried enabling “Write to disk” both before the column filter and inside the Learner-Node. This helped reduce RAM usage, but the learner node still gets stuck at 2%.

Attempts to Fix:

  1. Moved nodes into a new workflow to minimize RAM usage—didn’t solve the issue.

  2. Applied a column filter to remove the newly added columns—prevented crashing, but learner still stuck at 2%.

  3. Enabled “Write to disk”—reduced RAM usage but didn’t resolve the learner node being stuck.

Other Observation:

Whenever the learner node gets stuck or KNIME crashes, it happens right after the memory is dumped. This behaviour was the same BEFORE activating “write to disk”. I think the issue has obviously to do with some memory error. After this the RAM-Ussage stays below 16 GB yet the CPU-Usage is still high, but with many notable spikes. Usually when running the Learner-Node is was persistant at 100%. The KNIME application itself is barely responding to interaction.

Question:

Why does the learner node still seem affected by the new columns, even though I’ve filtered them out before loading the data into the H2O environment? What can I do to fix the issue?