Adjusting context window for local LLM using GPT4ALL nodes
I’m trying to get Llama 3.1 8B Instruct 128k work with KNIME on my MacBook Pro M4 machine. When testing this model directly from the GPT4ALL GUI I’m able to get it to summarise documents for me after increasing the model’s context length parameter in GPT4ALL. However, when I try to do the same using KNIME I get an error message that the prompt size exceeds the context window and I can’t find anywhere to configure the right context window size. There is a possibility to configure the response length, but that does is not helping. Is there a way to tell KNIME that the context length for this model is larger?