Model Uncertainty Margins in NFL Fourth-Down Decisions

## Key Finding Most reported "analytics controversies" are actually decisions made within the model's own margin of uncertainty. The model's preference is often weak, and the coach's choice is defensible. ## Brill, Yurko & Abraham (The American Statistician, 2025) - Standard approach maximizes estimated win probability but ignores estimation uncertainty - They use bootstrapping to report "boot%" (how often bootstrapped model repeats same recommendation) and 90% confidence intervals - **Only 48% of 2018-2022 fourth-down decisions were "confident"** (boot% ≥ 83%) - **27% were "uncertain"** (boot% below 67%) - **44% of all fourth downs had <1% win-probability difference between choices** - Analysts express far less uncertainty than the data supports - Bootstrap understates total uncertainty because it ignores model bias - Source: https://ryansbrill.com/pdf/statistics_in_sports_papers/Brill_Humility_TAS.pdf ## Wharton Model Example - Sept. 16, 2024: Fourth-down call with 90% CI for WP gain from going for it: [-5%, 8%] - Only 67% of bootstrapped models favored going for it - Described as "a lean" with little confidence - Contrasts with ESPN model which "leaned towards going for it" without stated interval - Source: https://wsb.wharton.upenn.edu/fourth-down-conversion-frequencies-are-higher-than-you-think/ ## Implications for Decision Auditing - Coaches should only be judged on plays where the model is confident or leaning - Since only 48% of decisions meet the confident threshold, a large share of second-guessing falls on plays the model itself considers uncertain - "Going against analytics" may often mean choosing within the margin of model uncertainty, not defying a clear recommendation - This reframes the analytics-vs-instinct debate: the question may not be which is better, but which is appropriate when the model itself is unsure ## Related: Cowboys Two-Point Attempt (Oct. 8, 2026) - nfl4th model: 14.5% win probability if kick, 13.7% if go-for-two - 0.8% gap — a toss-up, not a clear recommendation - Filed in media as "coach went against analytics" - Source: https://www.pascalinitiative.com/insights/bucs-cowboys-fourth-down-two-point/