Automated download of a series of files from the same server

Hello, Knimers,

I’m facing a problem that should be very easy to solve: I need to automatically download a series of (monthly) files from the same server, choosing files by inserting “start date” and “end date” of an arbitrary interval. These files are open and contain public data of the Brazilian national health system (from the servers of DataSUS, its IT Department). Such files are available from either one of two different sources: a) a “transfer page” from DataSUS, from where different (either individual or multiple) subsets of data can be selected using a series of six drop-down menus (at < Transferência de Arquivos – DATASUS >), but up to this moment, I only know how to do it manually; or b) an FTP server (where: Protocol: “ftp://”; Host: “ftp.datasus.gov.br”; Path: “/dissemin/publicos/SIASUS/200801/Dados/”; Generic File name: “PARS2403” (where “PA” stands for the subset of data; “RS” is the particular Brazilian state (from a total of 27), “24” represents the year (2024), and “03” is the month (March)); File extension: “.dbc” (a proprietary format for compressing microdata files); and so, this full generic url becomes: ftp://ftp.datasus.gov.br/dissemin/publicos/SIASUS/200801/Dados/PARS2403.dbc

Besides, Brazilian Health Ministry provides a free software (TabWin - that uses a graphic interface and is available at the same “trasfer page” cited above) to decompress DBC files (to DBF format) and to convert them into CSV format.

So, the (yet to be solved) task is: how could I automate the download of a series of X files using a workflow in Knime?

After trying (without success) several options (including “FTP connector + Transfer Files”; “GET Request + Binary Objects to Files”; etc.), I have been trying to use the following sequence of nodes:

Create Date&Time Range (for defining Start and End Dates, in the ‘Date&Time’ data type);

Cell Splitter By Position (to generate separate String columns for Year (YY), Month (MM) and Day (DD), all with 2 digits;

String Manipulation (to gather ‘data subset’, ‘state’, ‘year’, ‘mont’, and so to compose each file name in a row);

another String Manipulation (to gather ‘protocol’, ‘host’, ‘path’, ‘filename’, and ‘fileextension’);

a Table Row to Variable Loop Start (to define ‘files’ and ‘urls’ as flow variables);

External Tool (to access TabWin from inside Knime - but I could not reach to a full configuration of this node with TabWin operating from inside it);

a Loop End node to run all the loops for donwloading all files.

Here is an image of how far could I reach:

As you can see, the functionality of my workflow stopped at the External Tool node…

Can someone help me? Any alternative sequence of nodes, any functional workflow fits equally well to my needs (except using “R” or “Python”, which are currently far beyond my skills and computational resources).

Thank you for all your support. I really appreciate it.

Looking forward to your esponse.

Rogério.