Fleetidy — Crash Model - Machine Learning

Executive Summary

We built a machine-learning model to predict whether a specific truck (DOT + VIN power unit) will be involved in at least one reportable crash during Year 2, using only information available at the end of Year 1. The model was trained on 1.67 million power units with 270 engineered features drawn from FMCSA inspections, violations, and carrier census data.

We then pruned the full model down to a 20-feature production equation using bootstrap stability selection — trading a small amount of accuracy for a dramatically simpler, auditable formula.

0.6190

Full Model Test AUC

270 features • L1 regularisation

0.5889

Pruned Model Test AUC

20 features • L2 regularisation

−0.025

AUC Cost of Pruning

93% of features removed

1. Data & Target Definition

The prediction entity is a power unit, identified by the combination of a carrier’s DOT number and an individual vehicle’s VIN. This is the most granular level at which FMCSA inspection and crash records can be linked.

Target Variable

Label: ≥ 1 reportable crash in Year 2 (2025)
Positives: 38,136 units (2.28%)
Negatives: 1,633,503 units (97.72%)
Leakage control: no Year-2 data used as features

Feature Sources

Inspections: counts, OOS rates, violation types
Violations: severity weights, time decay, BASIC flags
Year-1 crashes: counts, severity, conditions
Census: fleet size, carrier type, mileage, state

Figure 1 — Dataset Scale and Class Imbalance

1.67 M power units with a 2.28% crash rate — a severely imbalanced rare-event problem

Train/Validation/Test split: 70% / 15% / 15% with stratified sampling to preserve the 2.28% positive rate in each partition. The test set was held out entirely until final evaluation.

2. Machine Learning Pipeline

The modelling pipeline has two stages: a full baseline model that uses all 270 features, and a pruned production model that uses only the 20 most stable and prevalent features.

1

Feature Engineering

Aggregated Year-1 inspection, violation, and crash events into unit-level numeric predictors: counts, sums, means, OOS rates, severity weights, time-decay scores, and carrier census fields. Result: 270 features.

2

Full Baseline Model

Linear logistic regression via SGDClassifier with L1 (Lasso) regularisation (α = 10⁻⁴) and balanced class weights. Trained on all 270 features.

3

Stability Selection

8 bootstrap runs on a stratified 400K-row subset. For each feature we measure selection frequency (how often it gets a non-zero coefficient), sign consistency (whether the coefficient direction is stable), and prevalence (fraction of units where the feature is non-zero).

4

Production Pruning

Kept only features with high stability and prevalence; excluded sparse one-hot indicators and missing-category flags. Tested k = 20, 25, 30 and selected k = 20 (best validation AUC). Refit with L2 regularisation.

5

Final Evaluation

Refit the selected 20-feature model on Train + Validation and scored the untouched Test set. Both full and pruned models evaluated on identical test partitions for a fair comparison.

3. Results — Full vs. Pruned Model

The table below compares the full 270-feature model against the pruned 20-feature alternative. The pruned model sacrifices 2.5 AUC points in exchange for removing 93% of features — a favourable trade for production deployment where simplicity, auditability, and governance matter.

Model	Features	Penalty	Validation AUC	Test AUC
Full Baseline	270	L1 (α=10⁻⁴)	0.6111	0.6190
Pruned Production	20	L2 (α=10⁻⁴)	0.5915	0.5889

Figure 2 — Test AUC Comparison

Full model vs. pruned production model on held-out test set

Figure 3 — Pruning Sweep

Validation AUC at different feature counts (k = 20, 25, 30)

Both models generalise well: validation and test AUCs are within 0.01 of each other, indicating no overfitting. The pruned model at k = 20 slightly outperforms k = 25 and k = 30, suggesting diminishing — and eventually negative — returns from adding more features beyond the stable core.

4. Stability Selection

Not all 270 features are equally reliable. Many are selected by L1 in one training run but drop out in another, or flip sign between runs. To find the robust core, we ran the model 8 times on different bootstrap samples and measured three stability dimensions:

270

Total features tested

32

Passed stability filters

20

Selected for production

Figure 4 — Feature Stability Landscape

Each dot is one of 270 features. X-axis: selection frequency across 8 bootstraps. Y-axis: sign consistency. Green dots = production candidates.

Only features in the upper-right quadrant — selected frequently and with consistent coefficient direction — are reliable enough for a production model. Most features cluster in the noisy lower-left, confirming the need for aggressive pruning.

Figure 5 — Stability Profile of the 20 Production Features

Selection frequency (blue) and sign consistency (green) across 8 bootstrap runs

5. The Production Equation

The final 20-feature model is a standard logistic regression — the crash probability is computed as:

P(crash) = sigmoid( −0.700 + ∑ coef_i × x_i )

The coefficients below show how each feature shifts crash probability. Blue bars indicate features that increase predicted crash risk; red bars indicate features that decrease it.

Figure 6 — Production Model Coefficients

Sorted by absolute magnitude. Positive (blue) = increases crash risk; Negative (red) = decreases crash risk.

What the Top Features Mean

Risk-Increasing Features

HOS driving measure (coef +3.06) — the single strongest predictor. Higher hours-of-service violation scores dramatically raise crash probability.
Authorised for hire (+0.84) — for-hire carriers have higher crash exposure than private fleets.
Driver OOS inspections (+0.27) — drivers placed out of service signal systematic fitness issues.
Violation time weights (+0.08) — recent violations count more than old ones.

Risk-Decreasing Features

HM violation sum (coef −0.52) — hazmat violations correlate with more cautious (lower-crash) operations.
Controlled substance measure (−0.20) — higher substance-testing activity is protective.
Driver fitness inspections w/ violations (−0.18) — more inspections catching issues = fewer crashes.
Vehicle maintenance violations (−0.11) — carriers flagged for maintenance tend to fix issues.

6. Performance by Data-Resolution Pool

Carriers differ enormously in how much inspection data is available. We segmented the 1.67M units into three pools based on DOT-level inspection volume and trained separate models for each. We also evaluated the global (all-population) model on each pool’s test split for comparison.

Pool	Units	Crash Rate	Pool Model AUC	Global Model AUC	Δ (Pool − Global)
High	670,842	2.75%	0.5580	0.5731	−0.015
Medium	755,762	2.06%	0.6436	0.6360	+0.008
Low	245,035	1.66%	0.6344	0.6653	−0.031

Figure 7 — Pool-Specific vs. Global Model AUC

Grouped bars: dedicated pool model (solid) vs. global model evaluated on same test split (striped)

The global model actually outperforms dedicated pool models for the high and low pools (by learning cross-pool patterns), while the medium pool benefits slightly from its own model. This suggests that a single well-regularised global model may be preferable to separate pool models for production simplicity. See the Pool Cross-Validation study for deeper analysis of pool-level variance.

7. Key Takeaways

1

Pruning Works

Removing 93% of features costs only 2.5 AUC points. The 20-feature equation retains the vast majority of predictive power while being simple enough to audit, explain to regulators, and deploy without complex infrastructure.

2

HOS Driving Is the Dominant Signal

The hours-of-service driving measure is by far the strongest predictor (coefficient 3.06, more than 3× the next largest). Fatigue-related violations are the clearest leading indicator of future crashes.

3

Stability Selection Reveals the Reliable Core

Of 270 features, only 32 survived stability filtering. Most features appear informative in a single model fit but fail to replicate across bootstrap samples. Without stability selection, the production model would rely on noise.

The Accuracy–Simplicity Trade-off

For underwriting and fleet safety workflows, the 20-feature model is the recommended choice. Its AUC of 0.589 means it correctly ranks a random crash unit above a non-crash unit 59% of the time — a meaningful edge over chance (50%) given the 2.28% base rate and the inherent stochasticity of crash events.

The full 270-feature model (AUC 0.619) is available for research and validation but is not recommended for production due to its reliance on unstable, sparse features that may not generalise to future time periods.

Crash Model — Machine Learning