Workflow log file rotation workaround

I usually run large and complex workflows that orchestrate many database reading & writing and data modifications performing nodes. In my typical workflow, tens of nodes execute either in parallel or in loops so I usually end up with hundreds of nodes executed at the moment my workflow finishes. In most cases it was hardly usable to solving problems investigating log file KNIME Analytics Platform produces since it doesn’t cover all the workflow.

I learnt KNIME writes log entries to a knime.log file until its size reaches 10MB. After that, it archives its content to a file knime.log.old.gz and continues writing entries to recreated knime.log file. Since it always re-write the archive file all former entries are lost. I have log level set set to DEBUG which is crucial for me if I want to find information I need. The conditions I’ve just described causes I never have complete log of whole workflow invocation.

What I would need is a possibility to configure log file rotation the way you know from logging framework. For instance in Log4J, you can set a size of single logging file (OK, 10M is OK) and number of old log files to keep, which is exactly what I miss in KNIME.

As a workaround I got used to forwarding the log file KNIME produces to a catch-all file using terminal command:

tail -F knime.log | tee knime.permanent.log

This ensures I have all log entries available after my workflow finishes.