Schema Converter (runtime.schema_converter)

Bidirectional conversion between SchemaDefinition and PySpark StructType.

Provides to_struct_type and from_struct_type for converting between the framework’s platform-independent schema model and PySpark’s native types. PySpark imports are deferred to function bodies so the module can be imported without a Spark runtime.

pyspark_pipeline_framework.runtime.schema_converter.to_struct_type(schema)[source]

Convert a SchemaDefinition to a PySpark StructType.

Parameters:

schema (SchemaDefinition) – The framework schema definition.

Returns:

A PySpark StructType with corresponding fields.

Raises:

ValueError – If any field type cannot be mapped to a PySpark type.

Return type:

StructType

pyspark_pipeline_framework.runtime.schema_converter.from_struct_type(struct_type, description=None)[source]

Convert a PySpark StructType to a SchemaDefinition.

Parameters:

  • struct_type (StructType) – A PySpark StructType.

  • description (str | None) – Optional description for the resulting schema.

Returns:

A SchemaDefinition with corresponding fields.

Return type:

SchemaDefinition