Using AI Agents for Dynamic Web Scraping and Data Enrichment in KNIME

Hi KNIME Community,

I’m experimenting with AI Agents in KNIME to orchestrate multiple tools dynamically based on user input. Here’s a concrete example I’m exploring:

Imagine a user wants to find Michelin-starred restaurants matching multiple criteria (cuisine type, city, price range, etc.). The workflow could be structured as follows:

1.

Predefined Scraping Tools – Multiple scraping workflows are already prepared for different sources. For example, one tool scrapes restaurants from Site A, another from Site B, etc. The AI Agent determines which tool(s) to execute based on the user’s request. An LLM could dynamically adjust filters and selectors within each scraping workflow to match the specified criteria.

1.

Data Enrichment Tool – For each restaurant, enrich missing data by querying other sources, such as Google Maps for addresses, financial databases for business status, or other public information.

1.

Deduplication & Merge Tool – Merge restaurant entries from multiple sources and remove duplicates to ensure a clean dataset.

1.

Response Generation / Export Tool – Format the final results and export them to CSV or present them in a user-friendly report.

My questions are:

*

How flexible are AI Agents in KNIME for modifying tool parameters or scripts on the fly, especially when an LLM needs to adjust scraping filters based on user input?

*

How are tools executed by the Agent? Can they be parallelized, or is execution strictly sequential?

*

Any best practices for managing complex scraping and enrichment workflows using AI Agents?

I’d love to hear from anyone who has experimented with similar AI-driven scraping and data enrichment workflows.

Thanks in advance!