Conceptual Question: Keeping History of Updated Values in SQLite - store as separate table?
I have a workflow that is working nicely in taking a stream of incremental data, transforming it, and then merging it with the DB Merge node to a SQLite DB Table that contains previously-pulled incremental data. Let’s call this Table One. It’s a great table. An excerpt of the workflow:
(Credit to @mlauber71 for helping me through my initial DB questions).
The dataset itself is a table of active real estate listings. However, a listing throughout its lifecycle will have price updates (seller decides to up/lower the list price), or other changes (new comments, updated dimensions, etc.). Each listing has a unique ID, called Listing Number, and I came to appreciate that I need a new table, which we should call Table Two.
Table Two, not yet created in any part of the workflow, should be a sort of “change tracker table” for every Listing Number. That way, I can pull up any Listing Number, and see all the changes that have happened. The native data set does include an “Updated Timestamp”. So Table Two could look like:
Listing# Updated Price SqFt Comments
001 Mon $300 500
001 Tue $350 500 Great!
002 Mon $600 1000
003 Mon $500
003 Wed $500 800
002 purposely left unchanged, in case that impacts the logic.
How would I go about efficiently creating a change tracker SQLite table?