加载中
正在获取最新内容，请稍候...
Great Expectations - 开源数据质量与验证工具
Great Expectations is the leading open-source tool for data quality, data profiling, and data documentation. It helps data teams eliminate pipeline debt and provides confidence when deploying new data projects.
Python
Added on 2025年6月29日10,504
Stars1,596
ForksPython
LanguageProject Introduction
Summary
Great Expectations is an open-source Python library for testing, documenting, and profiling your data to ensure quality and consistency across your data pipelines and workflows.
Problem Solved
Data quality issues are a major source of pain in data pipelines and analytics projects. Great Expectations addresses this by providing a principled way to test and validate data systematically, preventing 'pipeline debt' and ensuring data trustworthiness.
Core Features
Data Validation (Expectations)
Create verifiable assertions about your data, known as Expectations, such as `expect_column_to_exist` or `expect_column_values_to_be_unique`.
Automated Data Documentation (Data Docs)
Automatically generate rich, human-readable documentation about your data, validation results, and Expectations.
Data Profiling
Profile data automatically to learn about its structure, distribution, and unique values to help define Expectations.
Flexible Data Source Integrations
Integrates seamlessly with popular data processing technologies like Pandas, Spark, Dask, Snowflake, BigQuery, Redshift, and more.
Tech Stack
Python
Pandas
Spark
Dask
SQLAlchemy
使用场景
Great Expectations is useful in any scenario where you need to understand, validate, or document the quality of your data. Common use cases include:
场景一：数据湖/仓库入库前校验
Details
Automatically run data validation checks on data batches (e.g., daily loads) before they are committed to a data warehouse or data lake.
User Value
Prevents bad data from entering your storage layer, maintaining trust in your central data assets.
场景二：ETL/ELT流程中的质量控制
Details
Add data quality checks at various stages of your data transformation pipelines (e.g., after joining tables, before feature engineering).
User Value
Ensures data transformations are correct and intermediate data is clean, preventing errors downstream.
场景三：数据集文档自动化
Details
Generate living documentation for datasets that evolves with your data, making it easy for anyone to understand data structure and validation rules.
User Value
Improves collaboration and understanding across teams by providing up-to-date, accessible data documentation.
Recommended Projects
You might be interested in these projects
micrometer-metricsmicrometer
Micrometer is a vendor-neutral application observability facade that provides a simple metrics API over a variety of monitoring systems. Inspired by SLF4J, it allows instrumenting applications once and exporting telemetry data to multiple monitoring platforms.
Java
46431032
kubernetes-sigskustomize
Kustomize helps customize Kubernetes configurations declaratively without modifying the original YAML files. It's a standard part of `kubectl`.
Go
115242315
modelcontextprotocolservers
Open-source server implementations for the Model Context Protocol (MCP), designed to standardize how context data is managed and served for AI and machine learning models.
JavaScript
551406319