加载中
正在获取最新内容，请稍候...
Apache PyIceberg - Python API for Apache Iceberg Data Lakes
PyIceberg provides a Python library for interacting with Apache Iceberg tables, enabling data engineers and data scientists to manage and query large-scale, open data lake tables using Python.
Python
Added on 2025年5月22日746
Stars289
ForksPython
LanguageProject Introduction
Summary
Apache PyIceberg is the official Python client for the Apache Iceberg table format. It allows Python developers to programmatically work with Iceberg datasets, supporting operations like reading, writing, schema evolution, and partitioning.
Problem Solved
Working with traditional file formats or database systems for large-scale analytical data can be challenging due to lack of transactionality, schema evolution issues, and difficulty in managing large partitions. PyIceberg leverages the Iceberg format to provide reliable, high-performance data lake operations accessible directly within the Python ecosystem.
Core Features
Iceberg API Access
Provides native Python bindings to interact with Iceberg catalogs and tables.
Schema Evolution Support
Handles changes to table schema over time without requiring data rewriting.
Partitioning and File Management
Manages data partitioning and underlying data files efficiently.
Tech Stack
Python
Apache Iceberg
Arrow (pyarrow)
Data Processing Engines (Spark, Flink, Trino, etc. via integration)
使用场景
PyIceberg is ideal for various data engineering and data science workflows within a Python environment, including:
场景一：构建和管理Iceberg数据湖
Details
Use Python scripts to programmatically create Iceberg tables, add data, and manage table lifecycle.
User Value
Simplifies data lake setup and management using familiar Python tools.
场景二：Python ETL/ELT workflows
Details
Implement data transformation and loading processes directly in Python, reading from and writing to Iceberg tables.
User Value
Enables efficient data processing pipelines with transactional guarantees provided by Iceberg.
Recommended Projects
You might be interested in these projects
elebummRedditVideoMakerBot
A Python bot designed to effortlessly create compelling videos from Reddit posts with just one command, perfect for content creators and enthusiasts.
Python
78992046
microsofttorchgeo
TorchGeo is a PyTorch library providing datasets, samplers, transforms, and pre-trained models specifically designed for geospatial data, enabling researchers and developers to apply deep learning techniques to satellite and aerial imagery, and other spatial data types.
Python
3393437
localstacklocalstack
A fully functional local AWS cloud stack, enabling developers to develop and test their cloud & Serverless applications offline without the need for a live AWS connection. Accelerate development workflows and reduce testing costs.
Python
592534164