Python node differing batch size error

I’ve built a hacky python row filter, which seems to work fine for smaller tables, however for a little bigger table ~ 76k x 300 i run into a batching error.

The code is super simple:

import knime.scripting.io as knio
if len(knio.flow_variables["required_columns"]) != 0:
df = knio.input_tables[0].to_pandas()
mask = ~df[knio.flow_variables["required_columns"]].isin([-9999999]).any(axis=1)
knio.output_tables[0] = knio.Table.from_pandas(df[mask])
else:
knio.output_tables[0] = knio.input_tables[0]

An exception was raised by the Python Proxy. Return Message: Traceback (most recent call last): File “/home/user/software/knime5.2.5/bundling/envs/orgknimepythonscripting/lib/python3.11/site-packages/py4j/clientserver.py”, line 617, in callproxy returnvalue = getattr(self.pool[objid], method)(*params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/home/user/software/knime5.2.5/plugins/org.knime.python3.scripting.nodes5.2.4.v202405171011/src/main/python/knimescriptinglauncher.py”, line 237, in closeOutputs raise e File “/home/user/software/knime5.2.5/plugins/org.knime.python3.scripting.nodes5.2.4.v202405171011/src/main/python/knimescriptinglauncher.py”, line 231, in closeOutputs self.backends.teardownarrow(flush=checkoutputs) File “/home/user/software/knime5.2.5/plugins/org.knime.python3.scripting.nodes5.2.4.v202405171011/src/main/python/knime/scripting/backend.py”, line 345, in teardownarrow b.teardownarrow(flush and isactivebackend) File “/home/user/software/knime5.2.5/plugins/org.knime.python3.scripting.nodes5.2.4.v202405171011/src/main/python/knime/scripting/backend.py”, line 186, in teardownarrow self.writealltables() File “/home/user/software/knime5.2.5/plugins/org.knime.python3.scripting.nodes5.2.4.v202405171011/src/main/python/knime/scripting/backend.py”, line 173, in writealltables table.writetosink(sink) File “/home/user/software/knime5.2.5/plugins/org.knime.python3.arrow5.2.0.v202311290857/src/main/python/knime/arrow/table.py”, line 321, in writetosink sink.write(batch) File “/home/user/software/knime5.2.5/plugins/org.knime.python3.arrow5.2.0.v202311290857/src/main/python/knime/arrow/backend.py”, line 392, in write self.writebatch(data) File “/home/user/software/knime5.2.5/plugins/org.knime.python3.arrow5.2.0.v202311290857/src/main/python/knime/arrow/backend.py”, line 397, in writebatch raise ValueError( ValueError: Tried writing a batch after a batch with a different size than the first batch. Only the last batch of a table can have a different size than the first batch.