PySpark Pipeline Framework

Configuration-driven PySpark pipeline framework with HOCON configuration, resilience patterns, lifecycle hooks, and streaming support.

Build batch and streaming data pipelines using composable components, HOCON configuration files, and a rich set of operational features including retry policies, circuit breakers, data quality checks, audit trails, secrets management, and checkpoint/resume.

Note

Scala/JVM users may also be interested in spark-pipeline-framework, the Scala implementation using PureConfig and Typesafe Config.

Supported Versions:

• Python 3.10 – 3.13

• Apache Spark 3.4+ (optional runtime dependency)

Getting Started

• Getting Started
  • Requirements
  • Installation
  • Modules
  • Quick Example
  • Testing Components
  • Next Steps
• Scope & Design
  • Design Philosophy
  • What This Framework IS
  • What This Framework is NOT
  • When to Use This Framework
  • Architecture Decisions
  • Comparison with Alternatives
  • See Also

User Guide

• Components
  • Component Architecture
  • Creating a Component
  • SparkSession Access
  • Logging
  • Built-in Example Components
  • HOCON Configuration
  • Running a Pipeline
  • Dynamic Component Loading
  • Schema-Aware Components
  • Error Handling
  • Testing Components
  • See Also
• Configuration
  • Configuration Structure
  • Spark Configuration
  • Spark Connect Support
  • Pipeline Configuration
  • Component Configuration
  • Configuration Loading
  • Environment Variable Substitution
  • Pre-built Policies
  • See Also
• Configuration Validation
  • Quick Start
  • Validation Phases
  • ValidationResult
  • Common Validation Errors
  • Dry Run
  • CI/CD Integration
  • See Also
• Streaming Pipelines
  • Creating a Streaming Pipeline
  • Built-in Sources
  • Built-in Sinks
  • Example Pipelines
  • Running a Stream
  • Trigger Configuration
  • See Also
• Resilience
  • Retry Policy
  • Circuit Breaker
  • Combining Retry and Circuit Breaker
  • See Also
• Lifecycle Hooks
  • Available Hooks
  • Basic Usage
  • Hook Lifecycle
  • CompositeHooks
  • Custom Hooks
  • NoOpHooks
  • See Also
• Data Quality Checks
  • Basic Usage
  • Built-in Checks
  • Check Timing
  • Failure Mode
  • Custom Checks
  • See Also
• Audit Trail
  • Basic Usage
  • Audit Sinks
  • Audit Events
  • Configuration Filtering
  • See Also
• Secrets Management
  • Basic Usage
  • Providers
  • SecretsResolver
  • SecretsCache
  • Custom Providers
  • See Also
• Schema Contracts
  • Overview
  • Defining Schemas
  • Supported Data Types
  • Schema Validation
  • Schema-Aware Components
  • Spark Integration
  • See Also
• Checkpoint & Resume
  • Basic Usage
  • How It Works
  • Checkpoint Store
  • Pipeline Fingerprint
  • See Also
• Scala Migration Guide
  • HOCON Field Mapping
  • Example: Full Config Migration
  • API Migration
  • Key Conceptual Differences
  • See Also

Production

• Production Deployment
  • Packaging Your Application
  • Spark Connect Deployment
  • Databricks
  • Amazon EMR
  • Kubernetes
  • Configuration Management
  • Secrets in Production
  • Monitoring
  • Health Checks
  • See Also
• Contributing & Development
  • Development Setup
  • Running Tests
  • Type Checking
  • Code Style
  • Project Structure
  • Pre-commit Hooks
  • Running Examples
  • Pull Request Guidelines
  • See Also

Reference

• Architecture
  • Component Architecture
  • Batch Pipeline Flow
  • Streaming Pipeline Flow
  • Package Layout
• API Reference
  • Configuration (core.config)
  • Components (core.component)
  • Schema (core.schema)
  • Resilience (core.resilience)
  • Data Quality (core.quality)
  • Audit (core.audit)
  • Secrets (core.secrets)
  • Metrics (core.metrics)
  • Session (runtime.session)
  • DataFlow (runtime.dataflow)
  • Streaming (runtime.streaming)
  • Dynamic Loader (runtime.loader)
  • Schema Converter (runtime.schema_converter)
  • Runner (runner)
  • Examples (examples)
• Glossary

Indices and tables

• Index

• Module Index