加载中
正在获取最新内容，请稍候...
OpenLineage - 开源数据血缘元数据收集标准
OpenLineage is an open standard for collecting and analyzing data lineage metadata, enabling better data governance, compliance, and debugging across diverse data ecosystems.
Java
Added on 2025年6月12日1,965
Stars341
ForksJava
LanguageProject Introduction
Summary
OpenLineage is an open standard for collecting and analyzing data lineage metadata. It provides a common specification for capturing information about datasets, jobs, and runs, enabling interoperability and comprehensive lineage tracking across heterogeneous data environments.
Problem Solved
Lack of a common standard makes it difficult to track data lineage across different data processing systems and tools, hindering data governance and debugging efforts.
Core Features
Standardized Specification
Provides a vendor-neutral, open specification for collecting lineage events and metadata.
Extensive Integrations
Offers integrations with various data processing frameworks and tools to automatically capture lineage.
Tech Stack
Java
Python
API Design
JSON Schema
Kafka
Spark
Airflow
使用场景
OpenLineage is applicable in various scenarios where understanding and tracking the flow of data is critical:
场景一：数据管道血缘追踪
Details
Track the origin and transformation of data assets across various ETL/ELT jobs and data pipelines for auditing and debugging.
User Value
Quickly identify the source of data issues and understand the blast radius of potential changes or errors.
场景二：影响分析与变更管理
Details
Understand which downstream reports, dashboards, or applications will be affected by changes to upstream data sources or processing jobs.
User Value
Minimize risks associated with data schema changes or pipeline modifications by clearly seeing dependencies.
Recommended Projects
You might be interested in these projects
sveltejssvelte
Svelte is a radical new approach to building user interfaces. Whereas traditional frameworks like React and Vue do the bulk of their work in the browser, Svelte shifts that work into a compile step that happens when you build your app.
JavaScript
832784561
labmlaiannotated_deep_learning_paper_implementations
An extensive collection of annotated implementations and tutorials for prominent deep learning papers, covering transformers, optimizers, GANs, reinforcement learning, and more, designed to facilitate understanding through side-by-side notes.
Python
614106206
its-a-featureMythic
Mythic is an open-source command and control (C2) framework designed for red teaming and adversary simulation. It supports multiple users, agent types, and cross-platform operations, streamlining complex engagements.
JavaScript
3703477