What Is Column-Level Lineage?

Column-level lineage is the granular tracking of how individual data fields - specific columns in specific tables - are created, transformed, and consumed as they flow through data pipelines, warehouses, and analytical systems. Where table-level data lineage tells you that Table B was derived from Table A, column-level lineage tells you that the net_revenue column in the finance report was computed from gross_amount minus discount_amount in the orders table, after currency conversion using the fx_rate field from the exchange rate lookup.

This granularity is increasingly expected of organizations that operate in regulated industries, build AI models on enterprise data, or need to quickly assess the impact of schema changes across complex data environments.

Column-Level Lineage Defined

Column-level lineage answers a specific question for each field in each table: Where did this value come from, and how was it transformed to get here? The answer is a directed graph: source columns connect to transformation expressions, which connect to target columns, across every step in the pipeline.

A complete column-level lineage record for a metric like customer_lifetime_value might span:

• Source systems: raw events from a CRM, transaction records from an ERP, contract data from a document management system
• Transformations: aggregation logic in a dbt model, currency normalization in a Spark job, outlier filtering in a Python notebook
• Intermediate tables: staging tables, feature tables, aggregate tables
• Outputs: BI dashboard columns, ML training features, regulatory report fields

Each node in this graph is a specific column in a specific table at a specific point in time. The edges are transformation expressions - the SQL, Python, or Spark code that produced each output from its inputs. Getting this level of detail means parsing transformation code or recording column reads and writes at runtime, not just observing which tables data moves between.

Column-Level vs. Table-Level Lineage

Table-level lineage (also called dataset-level lineage) maps which tables were used to produce which other tables. It's the starting point for governance and is much easier to capture automatically - many orchestration tools and transformation frameworks log table-level dependencies. But table-level lineage has critical limitations:

Table-level lineage tells an auditor that customer data flows into a regulatory report. Column-level lineage tells them which fields - answering whether the email column (PII under GDPR) actually contributes to the output, or only a derived customer_segment. That distinction matters for the GDPR record of processing activities and for data minimisation. Table-level lineage cannot make that determination.

Regulatory Compliance Use Cases

No regulation names column-level lineage as such, but several set traceability expectations that are hard to meet without it:

• BCBS 239 - The Basel Committee's principles for effective risk data aggregation and risk reporting were written for global systemically important banks. Principle 3 expects accurate, reliable risk data aggregated on a largely automated basis, and the ECB's 2024 guide on risk data aggregation asks banks for complete and up-to-date data lineage at data attribute level, from data capture through extraction, transformation and loading. Column-level lineage is the practical way to provide that at scale.
• GDPR Article 30 - Records of processing activities must list, among other things, the purposes of processing, the categories of personal data and the recipients. Column-level lineage maps which PII fields flow through which pipelines to which outputs, which helps keep that record complete and current.
• SOX Section 404 - Management must assess internal control over financial reporting. Column-level lineage documents the transformation logic behind reported figures, an audit trail that is hard to keep current by hand.
• DORA - Article 8 requires financial entities to identify and document their information assets and to map the links and interdependencies between them. Lineage is one input to that map.

Impact Analysis

One of the highest-value practical uses of column-level lineage is impact analysis: determining what will break if an upstream schema changes. Without column-level lineage, the answer to "what is affected if we rename this column?" requires manual inspection of every downstream pipeline and report - work that takes days in complex environments and is error-prone.

With column-level lineage, the impact analysis is immediate: the lineage graph shows which captured downstream transformations reference the renamed column, which reports consume those transformations, and which AI models are trained on those features. For example, a change that might require reviewing 200 assets can be scoped to the 7 that actually reference the changed column.

How Column-Level Lineage Works

Capturing column-level lineage means resolving the SQL, Python, Spark, or dbt logic that produces each output, either by parsing it or by recording what the engine reads and writes at runtime. This is technically harder than table-level lineage, which can be captured from orchestration logs alone.

Three primary approaches exist, often used in combination:

• Static SQL parsing - Analyze SQL transformation code statically to extract column references and derivation expressions. Works well for straightforward SQL but requires a SQL parser that handles complex expressions, CTEs, window functions, and dialects (BigQuery SQL ≠ Snowflake SQL ≠ PostgreSQL SQL).
• Runtime execution logging - Instrument the execution of queries and transformations to capture actual column-level reads and writes. Captures dynamic SQL and code-generated transformations that static parsing misses. Higher infrastructure overhead. The OpenLineage standard defines a column-level lineage facet for this, which its Spark integration can emit.
• Platform-native metadata - Some platforms expose column-level lineage directly: Unity Catalog in the system.access.column_lineage system table, Snowflake in the ACCESS_HISTORY view (Enterprise Edition), and dbt in its Catalog on Enterprise plans. Reading these sources is the lowest-effort path for organizations using those tools. In Dawiso, for example, the Databricks connector reads system.access.column_lineage, while the Snowflake connector has no native column-to-column lineage and gets column lineage from SQL script parsing.

AI and Column-Level Lineage

The rise of ML and AI in enterprise analytics creates new requirements for column-level lineage. A model trained on enterprise data is only as trustworthy as the provenance of its training features. The EU AI Act classifies credit scoring of natural persons as high-risk (Annex III) and requires the data governance for such a system's training data to cover the origin of the data, data-preparation operations and an examination for possible biases (Article 10(2)). Column-level lineage is one way to show which raw data fields contributed to each feature in the training dataset.

Similarly, AI agents that query enterprise data - through MCP-connected tools or direct API access - should operate on columns that have known lineage and quality signals. An AI assistant answering "what was our net revenue last quarter?" should be able to verify that the net_revenue column it's retrieving has documented transformation logic and a passing quality score, not just that the table exists. Column-level lineage, surfaced through a data catalog, provides that context.

Conclusion

Column-level lineage is the highest-granularity form of data traceability available in modern data governance. Banking supervisors increasingly expect lineage at this level of detail, it supports data protection records, it is the enabling technology for precise impact analysis, and it is increasingly necessary for AI systems that must justify their outputs at the field level. Organizations that invest in column-level lineage infrastructure - through SQL parsing, platform-native metadata, or both - gain the ability to govern their data environments with a precision that table-level lineage alone cannot provide.

Sources

• Basel Committee on Banking Supervision, Principles for effective risk data aggregation and risk reporting (BCBS 239), January 2013 (Principle 3).
• European Central Bank, Guide on effective risk data aggregation and risk reporting, May 2024 (Section 3.4, data lineage at data attribute level).
• GDPR Article 30, Records of processing activities.
• 15 U.S.C. 7262, Management assessment of internal controls (SOX Section 404).
• Publications Office of the EU (EUR-Lex), Regulation (EU) 2022/2554 (DORA) (Article 8, Identification).
• Publications Office of the EU (EUR-Lex), Regulation (EU) 2024/1689, Artificial Intelligence Act (Article 10(2) and Annex III, point 5(b)).
• OpenLineage, Column Level Lineage Dataset Facet.
• Databricks, Lineage in Unity Catalog.
• Snowflake, Access History (column lineage, Enterprise Edition).
• dbt Labs, Column-level lineage.
• Dawiso Help, Sources of data lineage.
• Dawiso Help, Databricks connector.
• Dawiso Help, Snowflake connector.