Execute failed: 504 Executor did not send resource within 30000
Hi everyone,
we are running KNIME Business Hub 1.18.3 on a single-node Kubernetes cluster, with executors based on KNIME Analytics Platform 5.8.3. We often get this error in scheduled workflows that use the Call Workflow Service node:
The parent workflow is a scheduled deployment that calls other workflows on the same Hub through Call Workflow Service. The error happens mostly at night (for example around 02:45), when several scheduled jobs run at the same time on the same execution context. It is intermittent: the same workflow usually works when re-run manually. The node has about 126 GB of RAM. We recently had memory pressure and OOM kills, so we reduced the executor memory from 100 GB to 85 GB.
We would like to understand what “Executor did not send resource within 30000” exactly means. Is the called workflow not loaded in time by the executor? Is this 30-second timeout configurable, either on the Call Workflow Service node or on the Hub/execution context side? Could the error be caused by the executor being busy or under memory pressure when many jobs run concurrently? If so, is there a recommended limit of concurrent jobs per executor, or a best practice for calling workflows from scheduled jobs?
Thanks in advance for any help!