National Grocer Converts 5,100 SAS Jobs and 4.4 Million Lines to Open Apache PySpark on EMR and Glue

Executive Summary

A US grocery retailer with 2,400+ stores already ran a Hadoop leftover and a growing AWS analytics footprint (EMR, Glue, S3, MWAA). What they did not want was a second platform tax. SAS Grid plus DI Studio was still the system of record for replenishment, pricing, loyalty, and vendor scorecards — 5,100 scheduled jobs, 4.4 million lines of SAS, 2,100 autocall macros, and a 36-node Grid that was out of hardware support. MigryX parsed the estate and emitted Apache PySpark 3.5 (DataFrame API + Spark SQL) that runs on EMR Serverless for the heavy DAGs and AWS Glue for the short, high-cardinality jobs. Sixteen months later the Grid is powered off. Projected three-year savings: $7.4 million, almost all of it SAS license + Grid ops, not “cloud magic.”

Client Overview

Merchandising, replenishment, and loyalty analytics grew up in SAS because that is what the 2008 data-warehouse program standardized on. Enterprise Guide authored most analyst jobs. DI Studio owned the 1,180 nightly vendor and POS ingest flows. A smaller set of stored processes fed store-ops dashboards. The company had already moved raw POS and loyalty events to S3; SAS was still the place those events became decisions.

Platform engineering had evaluated Databricks and Snowflake. Both were rejected for this estate: the firm already paid for EMR capacity commitments, the security team had finished a Lake Formation rollout, and merchandising leadership would not accept a second catalog and a second IAM story. The requirement was explicit — open PySpark, their S3, their Glue catalog, their Airflow.

Business Challenge

• Job count is not program count. 5,100 Control-M entries pointed at 3,640 unique .sas programs plus 1,180 DI jobs. Many “jobs” were the same program with a store-region parameter. Treating them as 5,100 independent rewrites would have doubled the work.
• POS grain vs SAS MERGE. Nightly replenishment still used DATA-step MERGE with IN= flags across item, store, and vendor keys. Naive join translations broke “in SAS only” rows that merchandisers used as void-and-reclass signals.
• Macro-generated SQL. About 40% of DI mappings injected PROC SQL written by macros that read a merchandising calendar table at runtime. Static expansion needed that calendar as a fixture, not as a hope.
• Two runtimes, one codebase. Glue’s 15-minute idle and 10 DPU habits are wrong for a 4-hour chain-wide forecast. EMR Serverless is wrong for 2,000 three-minute vendor files. The conversion had to emit the same PySpark modules and a runtime hint, not two dialects.
• Hadoop muscle memory. Ops still thought in YARN queues. EMR Serverless application-level limits replaced those queues. Without a mapping, on-call would have treated every timeout as a parser bug.

The MigryX Approach

Inventory first: hash every program, collapse parameterized Control-M clones into one logical job with a parameter file, and build the DI metadata graph. That reduced the conversion surface from 5,100 entries to 3,640 programs + 1,180 flows — still large, but no longer padded.

The AST path is the same SAS parser used on Databricks programs. The emitter is not. Output is plain PySpark: spark.read / DataFrame transforms / spark.sql for PROC SQL, written as importable modules with a thin runner. No Databricks Workflow YAML, no Unity Catalog names, no dbutils. Table identifiers are Glue database.table. Storage is S3 prefixes that match the old SAS library names so merchandisers can find “PRD.LOYALTY” without a decoder ring.

Runtime placement is a classifier, not a preference:

• EMR Serverless — jobs whose 90th-percentile Grid CPU exceeded 20 minutes, or that shuffle more than ~80 GB (replenishment, loyalty identity, vendor compliance).
• Glue Spark — short jobs, many files, high fan-out (vendor ASN, planogram drops, 2,000+ store-level extracts).
• Spark SQL files — PROC SQL with no DATA-step state, registered as Glue jobs that only submit SQL. Those stay readable for analysts who will never touch a DataFrame.

MWAA DAGs were generated from Control-M + DI predecessors, then patched where the scheduler and the metadata disagreed (they did, on 94 flows). That patch list is part of the deliverable; pretending the graph was clean would have shipped a lying DAG.

Target Architecture

SAS Grid → MigryX → Apache Spark on EMR + Glue (no Databricks control plane)

On-premises

SAS Grid36 nodes, Control-M

DI Studio1,180 flows

EG + STPAnalyst jobs

→

MigryX

PySpark 3.5No dbutils

Runtime hintEMR vs Glue

→

AWS us-east-1

Amazon S3LIBNAME prefixes

EMR ServerlessHeavy shuffle

AWS GlueShort fan-out

MWAAControl-M graph

Lake FormationLoyalty PII

CloudTrailJob + table audit

This is Spark-the-engine, not Spark-the-product. If they later put Iceberg under these tables, the PySpark modules do not change — only the catalog writer. That was a design constraint, not a slide.

What we refused to emit

collect() to pandas for “simple” PROCs, rdd.map for DATA steps, and UDFs for SAS formats. Formats became broadcast lookup DataFrames (the 94-format catalog was 11 MB). DATA-step RETAIN became windows. The 71 programs that were genuinely sequential (point= loops over claim-like store incidents) stayed as mapPartitions with an explicit comment and a size cap — they are 71, not a license to write RDD code everywhere.

Estate Inventory and Cutover Waves

4.4M LOC includes macros once, not once per caller. DI XML is counted as generated SAS equivalent, because that is what the Grid actually ran. We do not count Control-M JCL as “code.”

Results

"We already knew Spark. We did not need a new logo on the architecture diagram. We needed 5,000 SAS jobs to become modules our on-call could grep. The parser did the MERGE cases we were going to get wrong by hand."

— Director of Data Platform, US grocery retailer