What Is Data Lineage?

Data lineage is the end-to-end record of how data moves from source systems, through transformations, to reports and AI models. It answers "where did this number come from?" for analysts and "show me every system that processed this data" for compliance officers.

As AI systems consume more enterprise data and regulators ask for more visibility into data processing, lineage has become essential infrastructure for any organization that takes data governance seriously.

What Is Data Lineage?

Data lineage documents the lifecycle of data from its creation or ingestion, through every transformation and movement, to its consumption in reports, AI models, and downstream systems. A complete lineage record shows which source systems a dataset came from, what transformations were applied, which teams and tools touched it, and where its outputs are used.

Lineage exists at three levels of granularity. System-level lineage shows that data flows from System A to System B to System C. Dataset-level lineage shows which tables and files are involved. Column-level lineage - the most detailed level, and the one audits and impact analysis usually need - shows exactly which source column contributed to which output column, and how it was transformed in between.

Why Data Lineage Matters

Data lineage addresses three problems that every data organization faces. Each is tractable with good lineage and nearly intractable without it.

Trusting analytical results

When a business leader asks "why did revenue drop 8% this quarter?", the data team first needs to verify whether the number itself is correct. Lineage makes that verification possible: follow the lineage from the dashboard metric back through the data warehouse to the source system, checking each transformation step. Without lineage, this investigation requires manual code review, conversations with multiple engineers, and days of effort - often longer than the analysis itself.

Impact analysis for data changes

When a source system changes - a column is renamed, a field definition shifts, an API endpoint is modified - data engineers need to know what downstream systems and reports will break. Without lineage, this impact analysis is guesswork. With lineage, the answer is immediate: the lineage graph shows the downstream dependencies it has captured, enabling engineers to plan changes and communicate impact before they happen.

Compliance and regulatory requirements

GDPR Article 30 requires controllers to keep records of processing activities, including the purposes of processing, the categories of personal data and the recipients, and lineage helps keep those records accurate as systems change. BCBS 239 expects banks to produce accurate risk data aggregated on a largely automated basis, and the ECB's 2024 guide on risk data aggregation asks for complete and up-to-date data lineage at data attribute level. SOX Section 404 requires management to assess internal control over financial reporting, and lineage shows how reported figures were produced. Each comes down to the same evidence: where data came from and how it was processed.

Types of Data Lineage

Data lineage can be captured and represented in several forms, each serving different audiences and use cases.

Technical lineage

Technical lineage captures the physical flow of data: which tables, columns, files, and systems are involved, and what operations (joins, aggregations, filters, transformations) are applied at each step. It is typically extracted automatically from SQL code, ETL pipeline definitions, orchestration tools, and API specifications. Technical lineage provides the detailed, machine-readable record that compliance and engineering teams need.

Business lineage

Business lineage translates technical lineage into terms non-technical stakeholders understand. Instead of "column rev_q3 in table fin_summary is derived from SUM(amount) WHERE type='revenue' in transactions," business lineage shows "Q3 Revenue is calculated from all transaction amounts classified as revenue in the source ERP system." Business lineage makes traceability accessible to analysts, auditors, and business leaders without requiring them to read SQL.

Column-level lineage

Column-level lineage is the most granular form, tracking the transformation of individual fields from source to destination. It is derived either by parsing SQL and transformation code or by engines that record field-level reads and writes at runtime, as Unity Catalog and Snowflake do. When an auditor asks "which source field does this calculated metric come from?", only column-level lineage provides a definitive answer.

Operational lineage

Operational lineage captures runtime behavior: when pipeline runs occurred, how long they took, whether they succeeded or failed, and how much data was processed. It extends the lineage picture from "what was the plan for moving this data" to "what happened when the pipeline ran." Data observability tools often produce this type of lineage as a byproduct of pipeline monitoring. OpenLineage, an open standard for lineage metadata, defines a common format for these run events, so schedulers and processing engines can report lineage to compatible backends.

How Data Lineage Works in Practice

Modern lineage platforms connect to the tools that process data - databases, ETL tools, orchestration platforms, BI tools, notebook environments - and automatically parse the transformation logic those tools contain. Rather than requiring engineers to manually document every transformation, lineage platforms extract that information from existing SQL, pipeline definitions, and code.

Once extracted, lineage is represented as a directed acyclic graph (DAG): nodes represent data assets (tables, columns, files, reports), and edges represent transformations and movements connecting them. Users navigate this graph forward ("what depends on this asset?") and backward ("where did this data come from?") through a visual interface.

Many data platforms now record lineage natively. Unity Catalog captures table and column lineage for queries run on Databricks and exposes it in the system tables system.access.table_lineage and system.access.column_lineage. Snowflake records column lineage in its ACCESS_HISTORY view and shows lineage in Snowsight, both in Enterprise Edition. Microsoft Purview and Google Cloud's Knowledge Catalog collect lineage from the services they integrate with, and the Power BI lineage view shows how a workspace's data sources, semantic models, reports and dashboards connect. Each graph is detailed for the systems it covers. A data flow that crosses several platforms needs those graphs joined into one view, which is the job of a catalog that reads lineage from each of them.

Use Cases

The value of lineage extends across organizational functions, each with specific scenarios that justify investment.

Data engineering: impact analysis

When data engineers modify a database schema, retire a table, or migrate a pipeline, lineage provides the dependency map to understand full impact. A migration project that would have required weeks of manual investigation can be scoped and planned far faster.

Data analysis: root cause investigation

When a report shows unexpected numbers, lineage gives analysts the ability to trace the metric back to source data and identify where the discrepancy was introduced. This "data detective" capability is one of the most practically valuable applications of lineage for business users.

Compliance: audit trails

Financial institutions use lineage to demonstrate BCBS 239 compliance. Healthcare organizations document HIPAA-compliant data handling. Privacy programs answer data subject access requests by showing which systems contain personal data and how it was processed. In each case, lineage replaces manual investigation with systematic, automated evidence.

AI governance: model documentation

AI governance requires documenting not just what models do, but what data they were built on. Lineage connects trained models to the datasets, transformations, and source systems that contributed to training data. For high-risk AI systems, Article 10 of the EU AI Act requires training data governance to cover data collection processes, the origin of the data and data-preparation operations. This documentation is essential for AI audits, debugging model performance, and demonstrating that AI systems are built on governed, compliant data. Metadata management adds the descriptions and quality scores that complete the picture.

Implementing Data Lineage

Successful lineage implementation requires thoughtful planning about scope, tooling, and integration with existing governance processes.

Start with critical data paths

Rather than capturing lineage for all data at once, start with the paths that matter most: datasets feeding regulatory reports, sources powering high-stakes AI models, tables that business-critical dashboards depend on. High-quality lineage for critical paths delivers more value than patchy coverage of everything.

Integrate lineage with your data catalog

Lineage is most valuable when integrated with a broader data catalog that provides business context for each node. Seeing that data flows from "Source System X" to "Table Y" is useful; seeing that it flows from "Customer Transaction System" to "Monthly Revenue Summary" - with descriptions, ownership, and quality scores attached - is far more useful.

Automate where possible

Manual lineage documentation is unsustainable in organizations where pipelines change frequently. Invest in tools that extract lineage from SQL code, ETL definitions, and orchestration tools automatically. Reserve manual effort for business lineage descriptions - the explanations of what each transformation does and why - that automated tools cannot generate. Active metadata platforms make this continuous collection the default.

How Dawiso Supports Data Lineage

Dawiso's interactive lineage gives both technical and business users a navigable view of how data flows at system, table and column level, with columns folded under their object until you expand them. The lineage diagram sits on the object's own page in the data catalog, so the page that shows where data comes from also carries its description, owners and business glossary links.

Dawiso builds lineage from four sources: lineage that supported connectors read from the source platform, transformation frameworks such as dbt, parsing of SQL scripts, and relations that users add by hand. SQL parsing is not on by default and is set up together with the Dawiso team. How deep lineage goes depends on the source. The Databricks connector reads column lineage from system.access.column_lineage. The Snowflake connector has no native column-to-column lineage, so column lineage there comes from SQL parsing. Power BI lineage connects reports, datasets, data sources and gateways at object level, and dbt lineage links models, sources and exposures.

Through the Model Context Protocol (MCP), AI agents can query Dawiso's lineage with the Dawiso MCP server, tracing upstream or downstream dependencies from a selected object alongside search and object details.

Dawiso treats lineage as a foundation for trust: when a business analyst uses data with clear, traceable lineage, they can rely on it with confidence. That confidence turns data from a source of debate into a foundation for decisions.

Conclusion

Data lineage connects every piece of data to its origins, its transformations, and its uses. Without it, data quality problems are mysterious, compliance is reactive, and AI models are built on foundations that cannot be fully understood. With it, organizations gain the visibility and accountability that effective governance requires.

The value compounds over time. Every piece of lineage captured makes impact analysis faster, compliance evidence more complete, and analytical trust more justified. Start with critical data paths, integrate lineage with your catalog, and automate wherever possible.

Sources

• GDPR Article 30, Records of processing activities.
• Basel Committee on Banking Supervision, Principles for effective risk data aggregation and risk reporting (BCBS 239), January 2013 (Principle 3).
• European Central Bank, Guide on effective risk data aggregation and risk reporting, May 2024 (Section 3.4, data lineage at data attribute level).
• 15 U.S.C. 7262, Management assessment of internal controls (SOX Section 404).
• Publications Office of the EU (EUR-Lex), Regulation (EU) 2024/1689, Artificial Intelligence Act (Article 10(2), data and data governance).
• OpenLineage, About OpenLineage.
• Databricks, Lineage in Unity Catalog.
• Snowflake, Access History (column lineage, Enterprise Edition).
• Snowflake, Data Lineage (Enterprise Edition).
• Microsoft Learn, Data lineage user guide for classic Microsoft Purview Data Catalog.
• Google Cloud, About data lineage.
• Microsoft Learn, Data lineage - Power BI.
• Dawiso Help, Data lineage.
• Dawiso Help, Sources of data lineage.
• Dawiso Help, Databricks connector.
• Dawiso Help, Snowflake connector.
• Dawiso Help, Power BI connector.
• Dawiso Help, dbt connector.
• Dawiso Help, Dawiso Model Context Protocol (MCP).