Fleetidy — Pool-Specific Cross-Validation Study

Executive Summary

We investigated whether motor-carrier crash-prediction models should be trained separately for carriers with different levels of FMCSA inspection history. By segmenting 1.67 million power units into high, medium, and low data-resolution pools we trained pool-specific logistic models and evaluated them with repeated stratified cross-validation (6 folds).

Key finding: models perform significantly better on low-inspection carriers (AUC 0.631) than on high-inspection carriers (AUC 0.563). Crucially, the high pool also exhibits the lowest fold-to-fold variance (σ = 0.012 vs. 0.028), indicating that crashes in heavily-inspected fleets are fundamentally stochastic — driven by exposure rather than detectable safety deficiencies. The low pool’s higher variance reflects genuine but noisy signal: when sparse-data carriers crash, they stand out.

0.563

High Pool AUC

670 K units • ± 0.012

0.626

Medium Pool AUC

756 K units • ± 0.028

0.631

Low Pool AUC

245 K units • ± 0.028

1. Introduction

The FMCSA Safety Measurement System assigns carriers to peer groups based on the number of inspections recorded over a rolling 24-month window. Carriers with many inspections produce dense feature vectors, while carriers inspected only once or twice yield sparse, noisy signals. We refer to this dimension as data resolution.

Rather than training a single model across the entire carrier population, this study asks: does separating carriers by data resolution and training pool-specific models improve crash-prediction accuracy? If the answer is yes, it would justify a stratified modelling pipeline where each pool receives its own regularisation and feature set.

Research Questions

Do mean AUC levels differ across data-resolution pools?
Is fold-to-fold variance meaningfully different?
Which pool benefits most from a dedicated linear model?

2. Data

The dataset is constructed at power-unit level (DOT + VIN) using a Year 1 → Year 2 temporal split: features are derived from Year 1 inspection, violation, and crash history, while the binary target indicates whether the unit was involved in a reportable crash during Year 2.

Each power unit inherits its carrier’s DOT number, which is then mapped to a data-resolution pool via a pre-computed assignment file. Pool boundaries are set by the number and recency of inspections recorded for each DOT.

Pool	Power Units	Crash Events	Positive Rate	Features
High	670,842	18,477	2.75%	265
Medium	755,762	15,598	2.06%	269
Low	245,035	4,061	1.66%	269
Total	1,671,639	38,136	2.28%	—

Figure 1 — Dataset Composition by Pool

Total power units and crash-positive units per data-resolution pool

Figure 2 — Year-2 Crash Rate by Pool

Percentage of power units involved in a reportable crash during Year 2

3. Methodology

Cross-Validation Strategy

Each pool is evaluated independently using RepeatedStratifiedKFold with 3 splits and 2 repeats (seed = 42), yielding 6 train/test folds per pool. Stratification ensures each fold preserves the pool-level positive rate, which is critical given the severe class imbalance (1.7–2.8% positive).

Model Specification

We use SGDClassifier with log-loss (equivalent to logistic regression trained via stochastic gradient descent). Class weights are balanced to up-weight the minority crash class. Each pool receives a tailored regularisation scheme:

Pool	Penalty	α	L1 Ratio
High	ElasticNet	1 × 10⁻⁴	0.50
Medium	L1 (Lasso)	1 × 10⁻⁴	—
Low	ElasticNet	1 × 10⁻⁴	0.50

Feature Standardisation

For every fold the feature matrix is z-score standardised using training-set statistics only. This prevents information leakage from the test set into the scaling parameters. Zero-variance columns are dropped before fitting.

Evaluation Metric

The primary metric is the Area Under the ROC Curve (AUC), computed on test-fold predictions after applying a sigmoid to the model’s raw decision function. AUC measures discriminative ability — the probability that a randomly chosen crash unit is scored higher than a randomly chosen non-crash unit. A value of 0.50 indicates no discrimination; 1.00 indicates perfect separation.

Logistic model:   P(crash | x) = sigmoid(w · z + b)

where   z = (x − μ_train) / σ_train     (per-fold standardisation)
        w, b = SGD-optimised coefficients
        penalty ∈ {ElasticNet, L1}           (pool-specific)

4. Results

Pool	AUC Mean	AUC Std	AUC Min	AUC Max	Range
High	0.5626	0.0124	0.5403	0.5724	0.032
Medium	0.6256	0.0277	0.5781	0.6483	0.070
Low	0.6314	0.0280	0.5763	0.6495	0.073

Figure 3 — Mean AUC by Pool

Bar heights show mean AUC; error bars show ± 1 standard deviation across 6 folds

The dashed line at 0.50 marks the random-chance baseline. All pools exceed it, confirming genuine predictive signal, but the gap between pools is substantial: the low pool outperforms the high pool by +0.069 AUC.

Figure 4 — Fold-Level AUC Distribution

Each dot is one cross-validation fold (6 per pool); horizontal bars show the pool mean

The high pool forms a tight cluster near 0.56 (low variance, low mean). The low and medium pools show wider spread but concentrate around 0.63–0.65, with occasional folds dipping to ~0.58 — still above the high pool’s best fold.

Figure 5 — AUC Range per Pool

Floating bars span each pool’s min–max AUC; diamond markers show the mean

The high pool has the narrowest band — it is consistently mediocre. The low pool has the widest range but its floor (0.576) still exceeds the high pool’s ceiling (0.572).

5. Discussion

The Central Finding: Stable Randomness vs. Noisy Signal

The most important pattern in this study is not just that the high pool has lower AUC — it is the combination of low AUC and low variance that tells the story. Meanwhile the low pool shows the mirror image: higher AUC paired with higher variance. These two facts point to a single coherent explanation.

High Pool: Crashes Are Fundamentally Stochastic

High-pool carriers are large interstate fleets with dense inspection histories and the highest crash rate of any pool (2.75%). Their crashes, however, appear to be driven primarily by exposure — more miles on the road, more hours behind the wheel — rather than by identifiable safety deficiencies that inspection-derived features can capture.

The data supports this directly: the model achieves an AUC of only 0.563 (barely above the 0.50 random-chance baseline) yet does so with remarkable consistency across all 6 folds (σ = 0.012, range 0.540–0.572). This is the signature of stable randomness. The model is not fluctuating between good and bad splits — it is consistently unable to separate crashers from non-crashers because the underlying crash process in this population is close to random with respect to the available features.

Put differently: when everyone in a pool looks similar in feature space (large fleet, many inspections, similar violation profiles) and crashes are a function of stochastic road exposure, no linear model — and likely no model at all without mileage or telematics data — will reliably distinguish who crashes next.

Low Pool: Real Signal, More Noise

The low pool tells the opposite story. These carriers have sparse inspection records and the lowest crash rate (1.66%), yet the model achieves the highest AUC (0.631). When a low-pool carrier does crash, it tends to stand out — its feature profile is distinctively different from non-crashers. The signal is real and the model finds it.

But the higher variance (σ = 0.028, range 0.576–0.649) reflects the smaller sample size (245K units vs. 670K) and the sparsity of the feature vectors. With fewer inspections per carrier, individual folds can shift noticeably depending on which carriers land in train vs. test. This is noisy signal — genuine predictive information that varies in strength from split to split because the underlying data is thin.

!

Low Variance ≠ Good Model

The high pool’s low variance (σ = 0.012) is not a sign of reliability — it is a sign that crashes are consistently unpredictable in this population. Meanwhile the low pool’s worst fold (AUC 0.576) still exceeds the high pool’s best fold (0.572), confirming that even noisy signal outperforms stable randomness.

Medium Pool: The Middle Ground

The medium pool (AUC 0.626, σ = 0.028) behaves almost identically to the low pool in both accuracy and variance, despite having three times the sample size (756K units). This suggests the boundary between “signal-rich” and “stochastic” crashes lies somewhere between the medium and high pools. Carriers with moderate inspection histories still exhibit enough heterogeneity for the model to exploit, while the most heavily-inspected carriers converge toward a homogeneous, exposure-dominated regime.

Production Implications

These results justify pool-specific modelling. A single unified model would be dominated by the high pool’s 670K rows and drag overall discrimination down. By separating pools we can:

Apply different regularisation strengths to match each pool’s data structure.
Assign pool-level confidence bands to risk scores, reflecting the different variances.
Focus feature-engineering efforts where they matter most — the high pool, which clearly needs richer or nonlinear features.

Limitations

The model family is limited to linear logistic regression. Nonlinear methods (gradient boosting, neural networks) may recover signal in the high pool that a linear boundary cannot.
Pool assignment is based on DOT-level inspection counts, which may conflate carrier size with data quality.
The study covers a single Year 1 → Year 2 window. Multi-year temporal validation would strengthen confidence in the pool-level patterns.

6. Conclusions & Next Steps

Conclusions:

Pool-specific performance differences are real and substantial (Δ AUC ≈ 0.07 between high and low).
The high pool’s combination of low AUC and low variance is the defining result: crashes in heavily-inspected fleets are fundamentally stochastic — driven by road exposure rather than safety deficiencies that inspection features can capture. The model is not unstable; it is consistently unable to distinguish crashers from non-crashers.
The low pool’s higher AUC with higher variance confirms that genuine predictive signal exists for sparse-data carriers, but the smaller sample and thinner feature vectors introduce fold-to-fold noise. This is noisy signal, not random chance.
The medium pool closely tracks the low pool in both AUC and variance despite being three times larger, suggesting the transition from “predictable” to “stochastic” crash dynamics occurs between the medium and high data-resolution tiers.

Next steps:

Exposure features — if the high pool’s crashes are exposure-driven, mileage or hours-of-service data (where available) may be the single most impactful addition for that tier.
Nonlinear models — train gradient-boosted trees per pool to test whether the high pool’s stochastic ceiling can be raised with interaction effects that a linear model cannot capture.
Feature importance comparison — run permutation importance per pool to confirm that the high pool’s features are genuinely uninformative rather than mis-specified.
Temporal validation — replicate on Year 2 → Year 3 to confirm the stochastic-vs-signal pattern is stable over time.