Data Lineage Visualization

SqlCAD includes a powerful lineage generation tool that allows you to visualize the dependencies between your data models. This helps in understanding data flow, debugging impact analysis, and documenting your architecture.

Generating Lineage

The lineage command scans your SqlCAD project and produces a visual representation of the graph.

sqlcad lineage [OPTIONS]

By default, this generates an interactive HTML file (lineage.html) using D3.js.

Visualization Modes

1. Entity-Level Lineage

This is the default mode. It shows the high-level relationships between Tables (Source), Entities (Silver), and Marts (Gold).

• Nodes represented by blocks (blue for Sources, orange for Entities, green for Marts).
• Edges represent dependencies (joins, subqueries).

Example:

sqlcad lineage

2. Column-Level Lineage

You can trace the provenance of a specific column back to its source roots. This is incredibly useful for impact analysis (e.g., "If I change the amount column in raw_orders, what Marts are affected?") or debugging (e.g., "Where does net_revenue come from?").

Example:

sqlcad lineage --column orders_mart.total_revenue

This will filter the graph to show only the paths that contribute to the total_revenue column in orders_mart.

Output Formats

HTML (Interactive)

The default output is a standalone HTML file containing an interactive D3.js graph.

• Zoom/Pan: Navigate large graphs.
• Tooltips: Hover over nodes to see details.
• Interactive: Good for exploration.

sqlcad lineage --format html --output my_graph.html

Mermaid (Static)

You can also generate a Mermaid.js flowchart definition. This is useful for embedding diagrams into Markdown documentation (like GitHub READMEs).

sqlcad lineage --format mermaid

Example Output:

graph TD
    source.orders --> silver.orders
    silver.orders --> gold.orders_mart

Understanding the Graph

• Layers: The graph automatically groups nodes by their layer (Source -> Silver -> Gold).
• Dependencies: Arrows point in the direction of data flow (from Source to Mart).