Fleetidy — Pool + Hazmat Cross-Validation Study

Executive Summary

Building on the pool-specific cross-validation study, we asked: does splitting carriers by hazmat status within each data-resolution pool reveal meaningful differences in crash predictability?

The answer is yes, with a two-part pattern. Hazmat cohorts show lower observed frequency (overall crash rate 2.105% vs 2.302%; violations per inspection 0.683 vs 1.139), but remain harder to predict than non-hazmat cohorts in every pool. Directionally, this supports a safer but more random-like hazmat profile in the current feature set.

0.531

High + Hazmat

120 K • ± 0.007

0.563

High + Non-Hazmat

551 K • ± 0.009

0.574

Medium + Hazmat

51 K • ± 0.032

0.630

Medium + Non-Hazmat

704 K • ± 0.014

0.571

Low + Hazmat

1.7 K • ± 0.046

0.626

Low + Non-Hazmat

243 K • ± 0.020

!

Hazmat Directional Profile: Lower Frequency, Lower Predictability

In every pool, hazmat carriers have lower AUC than non-hazmat carriers (gap +0.032 to +0.057). At the same time, hazmat shows lower raw crash and violation intensity overall. Together this supports the same directional interpretation used in the underwriting-randomness study: lower baseline frequency but more random-like outcomes within the current covariates.

High Pool Gap

+0.032

0.531 → 0.563

Medium Pool Gap

+0.057

0.574 → 0.630

Low Pool Gap

+0.055

0.571 → 0.626

Overall directional check: hazmat crash rate is about 8.5% lower (2.105% vs 2.302%), mean violations per unit are about 33.4% lower (1.2515 vs 1.8801), and violations per inspection are about 40.0% lower (0.6833 vs 1.1394). Weighted AUC is also lower for hazmat (0.544 vs 0.605), consistent with weaker separability.

1. Data & Segmentation

We subdivide the three data-resolution pools (high, medium, low) by hazmat endorsement status, defined as smsc_hm_flag ≥ 0.5 at the DOT-level context feature. This produces 6 subgroups, each receiving its own model and cross-validation evaluation.

Group	Power Units	Crash Events	Positive Rate	Features	AUC Mean
High • Hazmat	120,149	2,954	2.46%	248	0.5314
High • Non-Hazmat	550,693	15,523	2.82%	262	0.5635
Medium • Hazmat	51,267	672	1.31%	258	0.5739
Medium • Non-Hazmat	704,495	14,926	2.12%	267	0.6304
Low • Hazmat	1,662	18	1.08%	230	0.5712
Low • Non-Hazmat	243,373	4,043	1.66%	267	0.6259
Total	1,671,639	38,136	2.28%	—	—

Figure 1 — Population Composition

Power units per subgroup (log scale). Hazmat carriers are a minority in every pool.

Figure 2 — Crash Rate by Subgroup

Year-2 crash positive rate (%); hazmat carriers tend to have lower raw crash rates

2. Methodology

The methodology mirrors the pool-only study with one additional segmentation dimension:

Parameter	Value
Cross-validation	RepeatedStratifiedKFold (3 splits × 2 repeats = 6 folds)
Random seed	42
Model	SGDClassifier (log_loss, balanced class weights)
Segmentation	Pool (high/medium/low) × Hazmat (hazmat/non-hazmat)
Hazmat definition	`smsc_hm_flag ≥ 0.5` (DOT-level)
Metric	Area Under ROC Curve (AUC)

Each of the 6 subgroups is trained and evaluated independently. Features are z-score standardised within each fold using training-set statistics only. Pool-specific regularisation (ElasticNet or L1) is applied as in the parent study.

3. Results

Group	AUC Mean	AUC Std	AUC Min	AUC Max	Range
High • Hazmat	0.5314	0.0070	0.5237	0.5393	0.016
High • Non-Hazmat	0.5635	0.0094	0.5518	0.5776	0.026
Medium • Hazmat	0.5739	0.0321	0.5336	0.6257	0.092
Medium • Non-Hazmat	0.6304	0.0142	0.6035	0.6442	0.041
Low • Hazmat	0.5712	0.0464	0.5259	0.6527	0.127
Low • Non-Hazmat	0.6259	0.0205	0.5921	0.6486	0.057

Figure 3 — Mean AUC: Hazmat vs Non-Hazmat by Pool

Grouped bars compare hazmat (solid) and non-hazmat (striped) within each pool; error bars show ± 1 std

The dashed line at 0.50 marks random chance. Non-hazmat carriers outperform hazmat carriers in every pool. The medium pool shows the largest absolute gap (+0.057).

Figure 4 — The Hazmat Penalty

AUC difference (non-hazmat minus hazmat) per pool — how much discrimination is lost for hazmat carriers

Figure 5 — Fold-Level AUC Distribution

Each dot is one CV fold (6 per subgroup); horizontal bars show the subgroup mean

High-pool hazmat folds (leftmost cluster) form the tightest, lowest-performing group. Low-pool hazmat (green, far right) has extreme spread due to only 18 crash events total — some folds have just 6 positives in the test set.

Figure 6 — Stability vs Performance

Each bubble is a subgroup; x-axis = mean AUC, y-axis = fold-to-fold std; bubble size = number of crash events

The ideal position is bottom-right (high AUC, low variance). Non-hazmat subgroups cluster closer to this ideal. The top-left region (low AUC, high variance) contains the hazmat subgroups with small crash counts.

4. Discussion

Directional Interpretation for Underwriting

The data supports a practical directional assumption: hazmat cohorts in this dataset are lower-frequency on observed outcomes but more random-like for model discrimination. This is an associational finding (not proof of a single cause), and likely reflects a blend of operating discipline, regulatory standardization, and feature limitations.

Three factors likely contribute to the consistent lower-AUC hazmat pattern:

Regulatory homogeneity. Hazmat carriers face stricter DOT requirements (placarding, special routing, enhanced inspections). This narrows the variance in their inspection-derived features — they all look more similar in feature space, making it harder for a linear model to separate crashers from non-crashers.
Crash causation differs. Hazmat crashes may be driven more by route-specific hazards (mountainous terrain, urban congestion) and cargo-related factors (load stability, chemical reactions) that inspection features cannot capture.
Smaller sample sizes compress the signal. Hazmat carriers represent only 10–18% of each pool. With fewer crash events (as low as 18 in the low pool), the model has less material to learn from.

!

High + Hazmat: Near Random Chance

The high-pool hazmat subgroup achieves AUC 0.531 with σ = 0.007. This is the lowest-performing, most stable group in the entire study. At 120K units with 2,954 crashes, this is not a sample-size problem — it is a fundamental signal problem. In underwriting terms, this profile is compatible with a lower-frequency but random-like segment where pricing and portfolio controls can matter more than granular feature-based selection.

Non-Hazmat Carriers: Where the Signal Lives

The non-hazmat subgroups achieve the study's best discrimination: medium non-hazmat leads at 0.630 (704K units, σ = 0.014) and low non-hazmat follows at 0.626 (243K units, σ = 0.020). These carriers have more heterogeneous operations and inspection profiles, giving the model more variance to exploit.

Low-Pool Hazmat: A Data Warning

With only 18 crash events across 1,662 power units, the low-pool hazmat subgroup is statistically underpowered. Its AUC of 0.571 looks reasonable but the standard deviation of 0.046 — by far the highest in the study — means individual folds swing from 0.526 to 0.653. Any conclusions about this subgroup should be treated with caution.

Comparison with Pool-Only Results

The pool-only study reported overall AUCs of 0.563 (high), 0.626 (medium), and 0.631 (low). Splitting by hazmat reveals that these averages were pulled down by the hazmat subgroups. The non-hazmat portions consistently match or exceed the pool-level figures, while the hazmat portions consistently fall below them.

AUC Heatmap

A visual summary of mean AUC by pool and hazmat status. Darker colour = stronger discrimination.

Hazmat

Non-Hazmat

High Pool

0.531

Near random

0.563

Weak

Medium Pool

0.574

Weak

0.630

Moderate

Low Pool

0.571

Weak*

0.626

Moderate

* Low + Hazmat: only 18 crash events — treat with caution

5. Conclusions & Next Steps

Conclusions:

Hazmat carriers are directionally lower-frequency than non-hazmat carriers in this dataset (overall crash rate 2.105% vs 2.302%; violations per inspection 0.683 vs 1.139).
Hazmat carriers are also consistently harder to predict than non-hazmat carriers across all three data-resolution pools (AUC gap: +0.032 to +0.057; weighted AUC 0.544 vs 0.605).
The high-pool hazmat subgroup (AUC 0.531, σ 0.007) is the most resistant to prediction — near random chance with exceptional stability. This supports a random-like interpretation with respect to inspection-derived features.
Non-hazmat carriers in the medium and low pools offer the best discrimination (AUC 0.626–0.630), suggesting that crash prediction should prioritise these segments for operational deployment.
The low-pool hazmat subgroup is statistically underpowered (18 events, σ 0.046) and should not drive modelling decisions.
The hazmat split provides marginal but real improvement over pool-only models. Non-hazmat subgroups consistently match or exceed the pool-level AUC from the parent study.

Next steps:

Use the directional hazmat assumption in pricing — treat hazmat as lower baseline frequency but less separable with current features, and calibrate margins/thresholds accordingly.
Hazmat-specific features — incorporate routing data, cargo classification, and hazmat-specific inspection outcomes to improve discrimination for hazmat carriers.
Merge low-pool hazmat into medium — with only 18 crash events, the low hazmat subgroup may benefit from pooling with medium hazmat to increase statistical power.
Nonlinear models — test gradient boosting per subgroup to determine if interaction effects can recover signal in hazmat populations.
Cost-sensitive thresholds — given the different base rates and AUC levels, deploy subgroup-specific decision thresholds calibrated to each population's prediction quality.