SEMMA in Practice: When a Linear Regression Beat the Random Forest

SEMMA was created by SAS Institute to organise the work inside SAS Enterprise Miner. Compared with CRISP-DM it is deliberately narrow. It has no business-understanding phase and no deployment phase. It assumes the problem is already defined and covers the technical core, from data to an assessed model.

That narrowness turns out to be useful. It forces you to be rigorous about the parts most people rush: how you sample, and how you assess.

Problem: predict a person’s annual medical insurance charges from age, sex, BMI, number of children, smoking status and region.

Data: Kaggle Medical Cost Personal Dataset, 1,338 policyholders.

I mapped every step to the SAS Enterprise Miner node it represents. A SAS program (semma_insurance.sas) in the repo reproduces the flow in SAS OnDemand for Academics.

S: Sample

SEMMA’s first question is whether the data is representative, and how to partition it so the final assessment is honest.

Integrity first. There is one exact duplicate row. If it is left in, the same person could end up in both training and test data, which is a small but real leak. I removed it before partitioning.

Sampling decision. At 1,337 rows, down-sampling would only throw information away. So the real Sample decision here is the partition:

· 60% train / 20% validate / 20% test, in the style of the SAS Data Partition node

· stratified on smoker × charge quartile, because smokers (about 20%) drive the expensive tail

Proof of representativeness. I ran Kolmogorov–Smirnov and χ² tests comparing each partition to training. Every p-value is above 0.05, and the smoker share is 20.6% / 20.2% / 20.5% across the three partitions.

E: Explore (training data only)

Univariate distributions on the training partition: charges are right-skewed, log(charges) is symmetric

Charges by smoker, age and BMI: smokers jump in cost at BMI 30

Four findings shaped everything that followed:

1. charges is right-skewed (skewness 1.54). Its log is nearly symmetric (–0.08).

2. Smoking dominates, with Spearman ρ = 0.66 with charges.

3. Age adds a near-linear effect, which shows up as three parallel bands.

4. The key interaction. Smokers with BMI ≥ 30 average about $42,000, while smokers with BMI < 30 average about $21,700. For non-smokers, BMI barely matters ($8.3k vs $8.7k).

The “outliers” in charges are 97% smokers. They are real expensive cases, not errors, so I kept them.

M: Modify

Get Anshita Apps’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Each transformation traces back to an Explore finding:

All encoding and scaling sit inside a pipeline fitted on training data only.

M: Model

I fit six candidates, mirroring the SAS modeling nodes:

· Regression on raw inputs (to measure the value of Modify)

· Regression with Modify features, on dollars

· Regression with Modify features, on log(charges)

· Decision tree, random forest and gradient boosting, each tuned with 5-fold CV

A: Assess

Results on the validation partition, as in the SAS Model Comparison node:

Validation RMSE by model: the engineered linear regression wins

Three lessons sit in this table.

1. Modify did most of the work. Adding three engineered interaction features moved linear regression from R² 0.73 to 0.86. That was enough to beat both ensembles on RMSE, and the model stays fully interpretable. Each coefficient is a dollar effect an underwriter can read.

Permutation importance of the champion: the smoker interaction features dominate

2. Assess rejected a hypothesis that Explore suggested. The log target looked right because the distribution became symmetric. But it produced the worst model. The smoker effect behaves like a roughly fixed dollar surcharge, and a log-scale model distorts that. This is why SEMMA keeps Explore and Assess as separate steps: visual intuition has to be confirmed on held-out data.

3. Metrics disagree for a reason. Gradient boosting has the lowest MAE, so it is best on typical policies. But it makes bigger errors on the expensive tail, which RMSE penalises. For an insurer, mis-pricing the expensive tail is the costly mistake.

Champion on the untouched test set: RMSE $4,261, MAE $2,306, R² 0.875.

Mean predicted vs actual charges by predicted decile (test partition)

The decile chart confirms the model ranks risk correctly. Mean actual charges rise steadily across predicted deciles, and the top decile averages 3.1× the overall mean.

Example scores from the final model:

· 30-year-old non-smoker, BMI 24: $6,551

· the same person as a smoker: $19,449

· smoker with BMI 34: $39,533

Where SEMMA stops

SEMMA ends at Assess. The saved model and score function are the hand-off. Business framing and production monitoring are outside the method, which is exactly the gap CRISP-DM fills. A small Streamlit estimator in the repo demonstrates the hand-off.