Recurrent Problem: Reading text files with changing format and schema
In workflows were the source of the data is a user supplied file of unknown and changing format, it gets basically impossible to create a working workflow without imposing strict limitations on the files format. But the later is let’s say not easy for non-technical users.
This is related to
How to autodetect format for each csv file
However the proposed solution is rather complex (I would say if you need to use python then probably there should be some type of improvement to knime to address that issue) as it used python to read files. The issue is that this fails as soon as you want to deploy on the business hub as the python script can’t deal with knime file paths.
Is there any other idea how to make this work?
To get a changing schema so different columns and a changing format say from tab to csv or others. basically there should be a way to force and autodetect of the format at run-time.
EDIT:
And to make things worse it seems the line reader just removed tabs so it will fail for any tab separated files?