加载中
正在获取最新内容，请稍候...
Dagster - 数据资产开发、生产和观测的编排平台
Discover Dagster, the data orchestration platform designed for the development, production, and observation of data assets. Streamline your data pipelines and improve reliability.
Python
Added on 2025年5月30日13,266
Stars1,700
ForksPython
LanguageProject Introduction
Summary
Dagster is an open-source data orchestration platform built for modern data teams. It provides a unified programming model and development environment for managing the entire lifecycle of data assets, from definition and execution to observation and maintenance.
Problem Solved
Traditional data orchestration tools often lack developer ergonomics, robust testing capabilities, and built-in observability for data assets. Dagster addresses these issues by providing a developer-centric framework and integrated tools for building, running, and monitoring data pipelines and assets.
Core Features
Code-Defined Pipelines
Define data pipelines using Python code, allowing for strong typing and testability.
Integrated Observation
Gain visibility into pipeline runs, data lineage, and asset health through a powerful UI.
Data Asset Management
Manage data assets throughout their lifecycle, from development to production.
Tech Stack
Python
React
GraphQL
Kubernetes
Docker
SQLAlchemy
使用场景
Dagster is used across various industries and team sizes to manage diverse data workloads. Key use cases include:
构建和管理ETL/ELT流程
Details
Automate the extraction, transformation, and loading of data from various sources into data warehouses or data lakes.
User Value
Ensures reliable and repeatable data loading processes, improving data freshness and accuracy.
机器学习工作流编排
Details
Orchestrate complex machine learning pipelines, including data preprocessing, model training, and deployment steps.
User Value
Provides visibility and control over ML experiments and production model pipelines, improving reproducibility and monitoring.
数据资产依赖管理
Details
Define dependencies between data assets (tables, files, models) and trigger updates based on changes in upstream assets.
User Value
Simplifies the management of complex data dependencies and enables efficient, incremental updates.
Recommended Projects
You might be interested in these projects
apachefory
Apache Fory is a blazingly fast multi-language serialization framework leveraging JIT compilation and zero-copy techniques for unparalleled performance in data exchange and storage.
Java
3302279
jhyjsoup
jsoup is a Java library designed for working with real-world HTML. It provides a very convenient API for fetching URLs, parsing HTML, interacting with the DOM, using CSS selectors, and cleaning user-submitted HTML against XSS attacks. It's built to handle the messiness of web content encountered in the wild.
Java
111462229
quic-goquic-go
quic-go is a pure Go implementation of the QUIC protocol, providing a fast and reliable alternative to TCP for modern internet applications. It aims to offer low latency and multiplexing capabilities.
Go
107251384