Python script node error when executing in the workflow only with 38M rows, but runs fine at a fe...

Environment:

• Virtual machine with 880 GB RAM, Windows 11

• KNIME 5.2.3, Columnar Backend

• Anaconda installation

I have a python script that works in configuration mode with the full 37.7M samples, and in the workflow if I row-sample my data at 5%. But when I apply the full sample (about 37.7M rows (and up to 50M coming in a later version of the project) the python script gets to 70% and then fails with this error in the KNIME Console:

ERROR Python Script 3:393 An exception was raised by the Python Proxy. Return Message: Traceback (most recent call last):

• File “C:\ProgramData\anaconda3\envs\py3knime3\lib\site-packages\py4j\clientserver.py”, line 617, in callproxy*

• returnvalue = getattr(self.pool[objid], method)(params)

• File “C:\Program Files\KNIME\plugins\org.knime.python3.scripting.nodes5.2.3.v202403221416\src\main\pythonknimescriptinglauncher.py”, line 237, in closeOutputs*

• raise e*

• File “C:\Program Files\KNIME\plugins\org.knime.python3.scripting.nodes5.2.3.v202403221416\src\main\pythonknimescriptinglauncher.py”, line 231, in closeOutputs*

• self.backends.teardownarrow(flush=checkoutputs)*

• File “C:\Program Files\KNIME\plugins\org.knime.python3.scripting.nodes5.2.3.v202403221416\src\main\python\knime\scriptingbackend.py”, line 345, in teardownarrow*

• b.teardownarrow(flush and isactivebackend)*

• File “C:\Program Files\KNIME\plugins\org.knime.python3.scripting.nodes5.2.3.v202403221416\src\main\python\knime\scriptingbackend.py”, line 186, in teardownarrow*

• self.writeall_tables()*

• File “C:\Program Files\KNIME\plugins\org.knime.python3.scripting.nodes5.2.3.v202403221416\src\main\python\knime\scriptingbackend.py”, line 173, in writeall_tables*

• table.writeto_sink(sink)*

• File “C:\Program Files\KNIME\plugins\org.knime.python3.arrow5.2.0.v202311290857\src\main\python\knimearrowtable.py”, line 321, in writetosink*

• sink.write(batch)*

• File “C:\Program Files\KNIME\plugins\org.knime.python3.arrow5.2.0.v202311290857\src\main\python\knimearrow_backend.py”, line 392, in write*

• self.writebatch(data)*

• File “C:\Program Files\KNIME\plugins\org.knime.python3.arrow5.2.0.v202311290857\src\main\python\knimearrowbackend.py”, line 397, in write_batch*

• raise ValueError(*

ValueError: Tried writing a batch after a batch with a different size than the first batch. Only the last batch of a table can have a different size than the first batch.

Because the python script is successful in execution at 5% of sample, I am focused on issues of how KNIME and Python integration scale (which may or may not be the right lens).

I noticed by watching system resources it starts to write to disc prior to the error.I tried rebalancing KNIME.ini memory and Columnar off-Heap memory in a few ratios, but the error returns:

100Gb KNIME / 700 GB HEAP

200Gb KNIME / 600 GB HEAP

300Gb KNIME / 500 GB HEAP

Any other trouble shooting tips or clues?

Are there limits to the amount of data Python and KNIME can transfer between each other?