Delta Table Reader (Labs) returns fewer rows than Spark/Synapse for the same Delta table

Hi everyone,

I'm seeing a discrepancy between the row counts returned by the Delta Table Reader (Labs) node in KNIME and a Delta read in Azure Synapse Spark, and I'm trying to understand whether I'm missing something in the configuration or if this could be a limitation of the Labs node.

Delta table structure

I'm reading a Delta table from a folder similar to:

input_dataset/

├── deltalog/

│   ├── 00000000000000000000.json

│   ├── 00000000000000000001.json

│   └── ...

├── part-xxxxx.parquet

├── part-yyyyy.parquet

└── ...

In KNIME, the Delta Table Reader is pointed to: input_dataset/

(i.e., the root Delta table folder, not the deltalog folder).

Results:

When reading the same location in Synpase:

spark.read.format("delta").load(path).count(): I get 158 rows


When reading the folder as raw parquet:

spark.read.parquet(path).count(): I get 804 rows


However, the Knime Delta Table (Labs) returns: 77 rows.

What's interesting is that 77 matches the row count of the most recently written parquet file in that location.

Questions

  1. How does the KNIME Delta Table Reader (Labs) determine which parquet files to read?

  2. Does it fully process the Delta transaction log (_delta_log) in the same way as Spark?

  3. Is there a way to inspect or log the parquet files that the Delta Table Reader is actually reading (similar to Spark's df.inputFiles())?

  4. Has anyone observed cases where the Delta Table Reader returns the row count of only a subset of active parquet files?

  5. Are there any known limitations with specific Delta Lake protocol versions or table features?

My expectation is that KNIME and Spark should return the same row count when reading the same Delta table, so I'm trying to understand why Spark returns 158 rows while KNIME returns 77.

Any guidance would be appreciated.

Thanks!