Huh. I can't find the source that I read, but I did find this one
arguing the exact opposite (ie, agreeing with you):
Maybe I'm misremembering my source, or maybe he has a good counter-argument to this guy (two smart analytics people may come to opposite conclusions). But absent any other evidence, I will revert to my previous position of thinking teams should take more chances on 2nd and short.
"You should take a shot" on second and short has been orthodoxy for so long that I'm reflexively suspicious of it.
Here's one way to look at it; it would take time to really analyze, but looking at PFR's win probability for first quarter tied games, ball on second and 1 at your own 48, expected points is
2.203; ball on first and 10 at your own 49, expected points is
2.191.
Second and 1 at the opponent's 31, expected points
3.589, first and 10 at the 30,
3.577.
So it may be slightly advantageous to get nine yards on first down instead of 10 yards, possibly because it gives you the opportunity to have a bigger play on second down.
Results of plays on second and 1, ball outside opponent's 10 (1994-2019):
Passes (n=3618): 9.8 yards average, 55.61% first downs, 3.54% turnovers (high!), 4.64% sacks, 2.54% TDs.
Rushes (n=8476): 4.2 yards average, 78.8% first downs, 0.72% turnovers, 0.76% TDs.
So teams are roughly twice as likely to rush on second and one. Rushing is substantially more likely to result in a first down, substantially less likely to result in a negative play, and substantially less likely to result in a big positive play. All of which is fairly intuitive. I am surprised, though, at the big disparity in turnovers; you're almost
five times as likely to turn it over when passing rather than rushing on second and one. So you get roughly a three-fold increase in yardage and TD probability in exchange for a five-fold increase in turnover risk.
My intuition is that turnovers are so expensive that the tradeoff isn't worth it. But a lot more detailed analysis would need to be done.