BYTEHERDERTwo dashboards. Same metric. One said 4,812. The other said 5,067.
The question I was asked was "which one is right". I want to walk through how long it took to find out, because the answer turned out not to be a number.
First: are the queries the same?
The obvious explanation is that somebody wrote the filter differently. One dashboard counts something the other excludes, and you find it by reading the SQL side by side.
So I read the SQL side by side. They were not identical, but they were equivalent: one used a LEFT JOIN with a WHERE ... IS NOT NULL, the other an INNER JOIN. Same result set. I ran both against the same table at the same moment and got the same number out of each.
Same query, effectively. Different answers on the dashboards. So it was not the query.
Then: are they reading the same table?
They were. Same schema, same table, same database. I checked the connection strings.
At which point the question had to change, because I had eliminated both halves of "a dashboard is a query against a table". If the query is the same and the table is the same, the only thing left is when.
So: when did each one read it?
This is where it got interesting. One dashboard was materialised by a job at 06:00. The other refreshed on view.
That alone does not explain a gap of 255 records unless the source is still moving, and the source was a third-party register that, we had been told, published once a day at 05:00.
I asked for the ingestion logs. The collector ran at 06:00, pulled a file, loaded it, and overwrote the table. One row in the log per run: timestamp, record count, exit code. Nothing about the file.
What was actually in the file?
Nothing in the pipeline had kept it. The collector downloaded, parsed, upserted and discarded. The only trace that a given file had ever existed was a count in a log line.
So I set the collector to write the raw file to object storage before parsing, with the fetch timestamp in the key, and waited two days.
The register did not publish once a day at 05:00. It published a first cut at 05:00 and backfilled corrections throughout the morning, with the last amendments landing around 13:30. The 06:00 snapshot was real, complete for 05:00, and already out of date by mid-morning. The on-view dashboard was reading a table that had been overwritten by the following day's 06:00 run, so at 14:00 on any given day the two dashboards were reading data captured nine hours apart from a source that moves.
Both numbers were correct. They were answers to different questions, and neither dashboard said which question it was answering.
The thing that was actually broken
The bug was not in either dashboard. The bug was that the pipeline threw away the only information that could have settled the argument in five minutes instead of two days: what was fetched, and when.
UPDATE is the enemy here. Every overwrite destroys the ability to answer "what did we believe yesterday, and why". That capability feels like overhead right up until someone disputes a number, and then it is the only thing that matters.
What we changed
Three things, none of them clever.
The raw record stays. Every fetch writes the untouched response to object storage, keyed by source and fetch time. Storage is cheap. A gzipped daily register file is under a megabyte, so a year of daily snapshots costs less than the meeting we had about the discrepancy.
Capture time is a column, not a log line. Every derived row carries the fetch timestamp of the record it came from. Not the row's own updated_at, which tells you when your pipeline wrote it. The time the source was observed.
Transformations are steps, not an overwrite. Raw, normalised, derived. Each layer reads the one below and writes its own table. A wrong assumption in the normalisation is now replayable from the raw layer instead of requiring a re-fetch from a source that has since moved on.
Every figure on both dashboards now carries the capture time it was computed from, in the corner, in mono, looking slightly ugly. It has settled four arguments since.
Where the models go
This comes up on every data engagement now, so to say it plainly: language models are useful in this kind of system, and they do not go in the chain of custody.
Triage, summarising, suggesting which of three records probably describe the same entity, drafting a first pass at a taxonomy. All good, all things that save real hours.
But a model cannot be cross-examined. You cannot ask it, six months later, which input produced which output, and get an answer you could defend to somebody who was not in the room. So a model's output is a suggestion that a human or a deterministic rule accepts, and the acceptance is what gets recorded, with the suggestion stored beside it. The moment a model's judgement is written straight into the derived layer with no record of what it saw, the traceability is gone and you are back to two dashboards and an argument.
The short version
Anyone can pull a dataset and draw a chart. The chart is not the hard part and it is not the valuable part.
The valuable part is being able to answer "where did this number come from" in the second year, when the source has changed its schema, the analyst has moved on, and somebody who was not there is disputing the result. That capability is not something you add later. It is a property of how the pipeline stores things, and if the pipeline overwrites, it is already gone.
Provenance is not paperwork around the product. On this kind of system it is the product. The chart is just the part people look at.