mcpField
T
tracing-upstream-lineage

tracing-upstream-lineage

github.com/necatiarslanData & databases4446 installsUpdated 21w ago
Repository

About tracing-upstream-lineage

A skill for tracing upstream data lineage in Airflow, from a table, column, or DAG to its producing DAG and source systems. It uses DAG code, run history, connections, and logs to build a lineage report.

Key features

  1. 01Identify table, column, or DAG targets
  2. 02Find producing DAGs and inspect source
  3. 03Trace tables, systems, and files
  4. 04Review upstream run health and logs
  5. 05Produce a lineage report

Use cases

  1. 01Find the sources of a target table
  2. 02Trace a column through DAG transformations
  3. 03Check upstream DAG freshness and health

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/necatiarslan/airflow-vscode-extension --skill tracing-upstream-lineage

After installing, type/tracing-upstream-lineagein Claude Code to confirm it loaded.

Details

From the project README

Upstream Lineage: Sources

Trace the origins of data and answer "Where does this data come from?"

Lineage Investigation

Step 1: Identify the Target Type

Determine what we are tracing:

  • Table
  • Column
  • DAG
Step 2: Find the Producing DAG
  1. List DAGs: use list_active_dags and list_paused_dags
  2. Read DAG source: use get_dag_source_code
  3. If a run exists, use analyse_dag_latest_run to see tasks and logs
Step 3: Trace Data Sources

From the DAG code, identify source tables and systems:

  • SQL sources in FROM or JOIN clauses
  • External sources via operator hooks or connection IDs
  • Files in object storage

Use go_to_connections_view to inspect connection metadata.

Step 4: Build the Lineage Chain

Example:

TARGET: analytics.orders_daily
    ^
    +-- DAG: etl_daily_orders
            ^
            +-- SOURCE: raw.orders
            |
            +-- SOURCE: dim.customers
Step 5: Check Source Health
  • Use get_dag_runs or get_dag_history on upstream DAGs
  • For logs, use go_to_dag_log_view

Lineage for Columns

  1. Find the column in the target table schema
  2. Search DAG source for references
  3. Trace transformations and mappings

Output: Lineage Report

Include:

  • Summary of sources
  • Lineage diagram
  • Source details (connections, freshness)
  • Transformation chain
  • Data quality implications

Related Skills

  • checking-freshness
  • debugging-dags
  • tracing-downstream-lineage
  • annotating-task-lineage
  • creating-openlineage-extractors

Comments

Markdown supported