P&C Insurer Moves 2,700 SAS Programs and 2.1 Million Lines to Polars — Spark Was the Wrong Machine

Executive Summary

A US property and casualty carrier ran actuarial, reserving, and statutory reporting on a 16-node SAS Grid. The job count was the problem — 2,700 Enterprise Guide programs, 2.1 million lines, 860 macros — not the data size. Median working set was 3.4 GB. P95 was 38 GB. A Spark cluster would have spent more time in task scheduling than in compute, and the actuarial staff would have inherited executor-memory folklore they did not ask for. MigryX converted the estate to Polars LazyFrames (expressions + pl.scan_parquet) running as Kubernetes Jobs. Eleven months later the Grid is gone. Three-year savings: $3.6 million. The 41 programs that spilled past a 64 GB worker were not “forced into Polars”; they went to a small Spark fallback the parser already knew how to emit.

Client Overview

The analytics org is actuaries and statutory accountants, not a data-platform army. They live in SAS because PROC SUMMARY, PROC UNIVARIATE, and a 15-year reserving macro library are how triangle and indication work gets signed. Data arrives from a policy admin system and a claims warehouse as nightly extracts — wide, not web-scale. The Grid existed because SAS licensing was sold as a grid, not because 3 GB files needed 16 nodes.

IT had a standing offer from the enterprise data team: “put it on the Spark lakehouse.” A two-week spike of 40 programs showed what anyone who has profiled SAS EG jobs already knows. Startup + shuffle for a 2 GB join was slower than SAS on one box, and the error messages were worse. Polars was already in a few Python sandboxes. The question was whether 2,700 programs could become that, with lineage and a replay harness, without each actuary rewriting their book.

Business Challenge

• Thousands of jobs, medium data. Treating this like a petabyte migration would have been the expensive mistake. The architecture had to be a job factory, not a cluster.
• Reserving macros are a product. 860 macros, some four levels deep, encode triangle logic and statutory exhibit mappings. Expanding them into unreadable Python would have failed model-risk review.
• SAS missing and formats. Actuarial code depends on . propagating through arithmetic and on format catalogs for line-of-business and state. Polars nulls are not SAS missing. The emitter had to implement the rule, not hope.
• first./last. and RETAIN. Loss triangles are sequential by accident-year / development. These map to over() / shift(), not to group_by().agg() slogans.
• Model risk / SR 11-7 adjacent. Even as a P&C shop (not a Fed bank), internal audit required output equivalence and change control. “It’s faster in Rust” is not a control.
• 41 jobs over 64 GB. Honest split: Polars workers have a memory ceiling. Those jobs were classified up front, not discovered in production OOM.

The MigryX Approach

Discovery measured working-set size from Grid logs (real mem, not file size). That classifier is the architecture: ≤64 GB peak → Polars Job; above → Spark module using the same logical plan where possible. 2,659 programs went Polars. 41 went Spark. No third runtime.

The emitter prefers LazyFrame expressions over eager DataFrame and over Python loops. PROC MEANS/SUMMARY/UNIVARIATE become group_by + expressions. PROC SQL becomes Polars SQL or an expression plan when the SQL used SAS extensions. DATA-step MERGE with BY becomes join plus explicit coalesce rules that match SAS. Formats become a sidecar Parquet lookup joined once, cached on the worker.

Macros that were functions became Python functions with typed parameters. Macros that were include-and-mutate-global became explicit argument passing — that is the only “rewrite” style audit accepted. We did not emit a SAS macro emulator.

Orchestration is existing Airflow on EKS. Each program is a Kubernetes Job with a memory request taken from the Grid profile (p95 × 1.4, capped at 64 GB). That mapping is why the first month of production did not page the platform team every night.

Target Architecture

SAS Grid → MigryX → Polars workers on Kubernetes (Spark only if >64 GB)

SAS Grid

SAS 9.416-node Grid

EG programs2,700 actuarial

Working setMedian 3.4 GB

→

MigryX

LazyFrame2,659 programs

Spark fallback41 jobs >64 GB

→

Runtime

Amazon EKSOne Job / program

Amazon S3Parquet extracts

Polarsscan_parquet

AirflowSame calendar

Worker imageMem from Grid p95

Git + tagProgram hash

Polars is the default because the data fits. The 41 Spark jobs are listed in the runbook by name. That list is the difference between an architecture and a hope.

Expression map that survived actuarial review

Estate Inventory and Cutover Waves

Close week is still close week. Finance programs dual-ran two month-ends before Grid disable. Reserving dual-ran one quarter. We did not “go live on a Friday.”

Results

"If you give actuaries Spark they will ask you to debug an executor. If you give them a Python module that prints the same triangle, they will sign it. Size the runtime to the file, not to the vendor deck."

— Chief Actuary, US P&C carrier