What is data lineage? The transformations behind a result

A revenue dashboard can show one number while the path behind it has already changed. The input was refreshed. A join key was remapped. A filter moved from the query into a manually maintained sheet. The number still has a label, but another analyst cannot tell whether it is the same object as last month.

Data lineage is the trace from a declared output or decision back through the input versions, transformations, joins, filters, aggregations, and downstream uses that produced it. It is a reconstruction boundary, not a badge of truth.

The revenue event-schema article owns the design of immutable commercial events. The evidence-review article owns the question of what evidence deserves to survive a decision. This page owns the path between a data object and the result that consumes it.

What does data lineage mean?

The word lineage is often used for a catalog of tables, a list of upstream systems, a warehouse feature, or a diagram on a wall. Those can be useful representations. They are not yet a reproducible trace unless the objects, versions, operations, and output use are identified.

The World Wide Web Consortium’s PROV-DM model gives a useful general vocabulary. It represents provenance through entities, activities, and agents, and includes relations such as usage, generation, derivation, association, and attribution (World Wide Web Consortium, 2013). The standard gives us a way to describe what participated in producing an object. It does not certify that a source or transformation was correct.

The practical distinction is:

Provenance is the broader context. Lineage is the applied path. Data quality is the fitness judgment. Event history is an observation record. One does not substitute for the others.

Which fields make a lineage trace reproducible?

Start with the result, not the diagram. Write down the unit and decision first, then preserve the objects that connect the result to its inputs:

1. Output: the field, table, metric, or decision object being explained.
2. Input: each source object, its owner, and its exact version or snapshot timestamp.
3. Operation: the filter, standardization, join, aggregation, calculation, or manual edit.
4. Key: the field or compound key used to connect records, including its uniqueness test.
5. Clock: event time, ingestion time, processing time, and the reporting cutoff where they differ.
6. Validation: the check that the operation produced the expected row count, grain, or boundary.
7. Use: the dashboard, report, model, or decision that consumes the output.
8. Exception: late data, failed check, changed rule, manual override, or unresolved mismatch.

The output needs a version too. A current table is not a stable object merely because its name stayed the same. A time-stamped snapshot can be compared with another snapshot. An event stream can answer a different question about what happened between them.

What does a lineage graph look like?

The following six rows are synthetic. They show the level of detail needed to reconstruct a small result. They are not warehouse records, customer records, or a current company’s data.

The table does not claim that the output is right. It makes the claim inspectable. L-02 is held because the expected key relationship has not yet passed. L-05 records a manual operation rather than hiding it in a notebook. L-06 shows that a late input can invalidate a previously issued result and reopen the decision.

Where can data lineage fail?

The common failures are structural:

• Snapshot substitution: a current table is compared with a prior event population without a shared cutoff.
• Join multiplication: a many-to-many join creates extra rows while the metric keeps the old denominator.
• Key drift: a renamed account, product, or territory breaks the path between versions.
• Late-arriving data: backfill changes a historical result but no invalidation or reissue is recorded.
• Hidden manual work: a spreadsheet correction changes the output without an owner, rule, or version.
• Unobserved filter: a query or dashboard applies a filter that the metric definition does not mention.

A lineage graph can expose each break. It cannot repair a bad source, decide whether a proxy measures the intended construct, or turn a descriptive result into a causal effect.

How should a team review a reported result?

Ask the owner to replay one result from the output backward:

1. Name the output, unit, reporting cutoff, and decision.
2. Resolve every input to a held version or timestamped snapshot.
3. Write each operation in order, including manual edits and exclusions.
4. Test every join key for expected grain and cardinality.
5. Reconcile input and output counts, totals, and boundary changes.
6. Mark late data, failed tests, superseded versions, and unresolved edges.
7. State whether the result is reproducible, held, reissued, or not reconstructable.

If the path cannot be replayed, narrow the result to the portion that can. If it can be replayed, keep the source-quality, measurement, causal, and transfer questions separate. Lineage answers how a result was produced. It does not answer whether the result is true or what caused it.