Global Manufacturer Converts 6,400 SAS Jobs and 5.4 Million Lines to Apache Iceberg — Same Tables on AWS, Azure, and GCP

Executive Summary

A diversified manufacturer (industrial equipment, after two acquisitions) had three clouds and one SAS problem. Plants and MES extracts already sat on AWS. Corporate finance and a Synapse estate sat on Azure. The acquired electronics division sat on GCP. Each region had its own SAS Grid. Combined: 6,400 scheduled jobs, 5.4 million lines, 2,400 macros, three format-catalog cultures. The board would not pick Databricks, Snowflake, or BigQuery as the storage standard — that fight had already wasted a year. They picked Apache Iceberg as the table contract. Compute could stay native: EMR/Glue and Athena on AWS, Synapse Spark and Fabric on Azure, Dataproc and BigQuery external tables on GCP. MigryX converted the SAS estate to Iceberg-writing PySpark (and Trino SQL where the job was scan-and-aggregate). Twenty months. Three catalogs. One spec. $9.2 million three-year gap versus three SAS enterprise renewals plus three Grids.

Client Overview

SAS grew up next to plants: yield, warranty, supplier quality, then finance consolidations after each acquisition. Nobody was going to move a German plant’s historians off AWS, or convince corporate treasury to leave Azure, or replatform the GCP division before the earn-out. A single vendor lakehouse was politically dead. An open table format was not.

The constraint we took as real: same Iceberg spec (v2, hidden partitions, snapshot isolation), different catalogs, no cross-cloud query in v1. Cross-cloud copies are explicit jobs, not “the mesh will handle it.” That sentence saved more time than any parser feature.

Business Challenge

• Three Grids, three calendars. “Fiscal week” was not the same object in US plants, EU finance, and the GCP division. Partition specs had to be written down per domain or Iceberg hidden partitions would just hide a mess.
• Catalog is the lock-in, not Parquet. Glue, Hive/Purview, and BigLake Metastore do not share a catalog. The conversion emits Iceberg table DDL plus a catalog adapter. Pretending one REST catalog existed on day one would have been fiction — they did not have Nessie/Polaris in production yet.
• SAS libraries were the schema. Thousands of .sas7bdat files with overlapping names. Iceberg table names are {cloud}_{domain}_{lib}_{member} until a human stewards a better name. We did not invent a beautiful enterprise model in the parser.
• Schema evolution is a feature and a foot-gun. SAS programs add columns by accident. Iceberg will accept that. Governance required a destage: additive columns in bronze, reviewed promotion to silver.
• Time travel vs SAS generation numbers. Finance wanted “the file from last Tuesday’s close.” That maps to a snapshot id recorded in the job log, not to a hope that overwrite-by-date is enough.

The MigryX Approach

One inventory across three Git remotes. Same AST, same PROC maps, same equivalence harness. Three emitters only where the catalog writer and the orchestrator differ. PySpark modules are cloud-agnostic except for a 40-line catalog.py per cloud (Glue vs HMS vs BigLake). That is the whole multi-cloud story in code terms.

Partition specs came from Grid scan patterns, not from a textbook:

• Plant yield / MES: plant_id + days(event_ts)
• Warranty: bucket(16, vin_or_serial) + months(claim_dt)
• Finance actuals: identity(ledger) + identity(fiscal_period) — not date, because their period is not a calendar month

Jobs that were PROC SQL + MEANS only became Trino (AWS Athena, Azure Trino-on-AKS, GCP BigQuery external / Dataproc Trino where already present). DATA-step and DI flows stayed PySpark writers. Compaction is a separate DAG (rewrite-data-files weekly on hot tables). We did not leave 10k tiny files because “Iceberg handles it.”

Shared Conversion Plane

Three SAS Grids → one Iceberg v2 spec → native catalog per cloud

Three Grids

AWS plants2,480 jobs

Azure HQ2,150 jobs

GCP division1,770 jobs

→

MigryX

Iceberg DDLv2 + partitions

PySpark writercatalog.py × 3

→

AWS

Amazon S3plant / warranty

Glue CatalogLake Formation

EMR ServerlessWriters

AthenaSQL path

Azure

ADLS Gen2finlake

Hive MetastoreSynapse + Fabric

Synapse SparkWriters

Microsoft PurviewScan + STTM

Google Cloud

Cloud Storagesupply / field

BigLakeExternal Iceberg

DataprocWriters

ComposerDivision DAGs

6,400 jobs / 5.4M LOC is the sum of three estates after de-duplicating 180 programs that existed in two clouds as copies. Those copies became one module and two catalog registrations — not two conversions.

AWS Architecture — Plants and MES

MES files land every few minutes. Iceberg commits are batched to 5–15 minutes on hot plant tables so the catalog is not a commit storm. Athena is for analysts; writers stay on EMR so a fat join cannot take the SQL warehouse hostage.

Azure Architecture — Corporate Finance

Finance refused “date=” partitions. fiscal_period is an identity partition. Time travel for close is “snapshot recorded in the close ticket,” which is a column on the audit table, not a story about Delta vs Iceberg.

GCP Architecture — Acquired Division

BigQuery users in the division query Iceberg in place. They do not get a second copy “for BQ.” The few consolidations that must land in Azure are Composer → authenticated copy jobs of specific snapshots, not a standing multi-cloud lake.

Estate Inventory

20 months is three waves per cloud, sequenced so finance close (Azure) never overlapped a plant cutover (AWS). GCP went last because the earn-out team could not take risk in year one. That is a calendar, not a tool limitation.

Results

"We were not going to win a platform war. We needed a table format the plants, treasury, and the acquired division could all write. Iceberg is that. The conversion had to respect three catalogs or it would have been another strategy deck."

— VP of Enterprise Data, global industrial manufacturer