Saturday, August 22, 2009

"Dubious Fielding Metrics"

In this week's chat, Tom Boswell had this to say about advanced fielding metrics while discussing Adam Dunn:
Those who think he is a terrible outfielder because of dubious fielding metrics (that, imo, are often wildly inacurate) are just wrong. Nobody gives up astronomical numbers of runs in LF -- 30-40+ runs a season. It's silly. There aren't enough at-the-margin plays out there to make that much difference. How can you possibly watch the game for years, have a sense of the range of an "average" outfielder, and think that Dunn (or anybody else) has that much negative impact?
Boz is referring to stats like Ultimate Zone Rating, Probabilistic Model of Range, John Dewan's plus/minus system, and Baseball Prospectus's Runs Above Average. I keep permanent links to most of these stats on the right hand bar of this page for reference. These stats are imperfect and have a larger margin for error than stats like OBP or SLG. But these stats have proven to have real predictive value, especially given adequate sample sizes.

Why is it so hard to believe that a left fielder can cost his team 30-40 runs? Let's look at some more familiar stats to establish a ballpark range of what's reasonable.

An outfielder can prevent runs basically three ways: range, throwing arm, and errors. Errors are the most familiar source of "lost runs," so let's start there. The left fielders league-wide in 2008 made 162 errors for an average of 5.4 per team.
Detroit, the team with the most errors in LF in 2008, committed twelve. Milwaukee was the best with zero.

Tom Tango has calculated the run value of each batting event, and the average error costs .506 runs, assuming a five-run-per-game scoring environment. MLB is at just 4.64 runs per game this year, so the current run value of an error is probably closer to .475 run.

Then again, errors by left fielders tend to be more costly than errors by infielders (for obvious reasons). For instance, the run value of a double, the de facto outcome of many outfield errors, is .781 run. Let's throw a reasonable dart and call it 0.575 run per LF error. That would mean the difference between the most and fewest errors among left fielders (12 * 0.575) was worth about seven runs in 2008.

Now let's look at throwing arms. League-wide, there were 283 assists by left fielders, or 9.4 per team. The league-leader was San Francisco with 16, and the teams with the fewest were San Diego, Tampa Bay, and Pittsburgh with five.

Tango doesn't list the run value of an outfield assist, but the value of an out is .304. Outfield assists tend to be particularly valuable outs, however, so let's round up to .4. That means that the range from best to worst LF arms is about 4.5 runs.

Outfield assists are just part of the story when it comes to preventing runs with throws. Strong arms and good throws also prevent runners from even trying to advance.

The only way to track these events is to look at the play-by-play data, and The Hardball Times has done just that. By their count, almost exactly the same number of runs were prevented by holding runners as throwing them out among qualified LFs in 2008 (1510 "kills" vs. 1517 "holds"). Assuming they're right, let's double the value of LF arms to nine.

Finally, we come to range, where the biggest difference between good and bad fielders exists.

In 2008, Kansas City led baseball in LF put-outs with 368. Philly and Toronto tied for the fewest put-outs at 278.
Therefore, assuming 0.575 runs per out, the gap between the best and worst teams in LF range is worth as much as 90 outs and 52 runs per season.

These rankings make sense, since David DeJesus, an excellent fielder, got the majority of the innings in LF for KC that year, while Pat Burrell and Adam Lind, neither good fielders, played the most innings for Philly and Toronto.

Of course, put-outs are also affected by the flyball rate of the pitching staff. Indeed, KC pitchers had among the higher flyball rates in the league at 38.6%. As a result, Royals outfielders saw 1573 flyballs all season, and DeJesus and the other left fielders converted 23.4% of them into outs.

Meanwhile, Toronto and Philly were towards the bottom in flyball rate at 33.5% and 33.8%, respectively. Total, Philly's OFs saw 1360 flyballs and Toronto's saw 1345. Philly LFs converted 20.4% of outfield flyballs into outs and Toronto LFs converted 20.7%.

We also know that the average team made 307 put-outs in left in 2008 in 1428 outfield flyballs with the average LF converting 21.5% into outs.

All this shows is that, yes, David DeJesus and friends got more chances than average, and Pat Burrell and Adam Lind had fewer. But even if the Phillies, Blue Jays, and Royals all had average numbers of opportunities, the Royals still would have gotten about 43 more outs from their superior range.

So let's assume then that over the course of the season, the gap from the best and worst range among left fielders is 43 outs. Further, let's assume that the run value of these additional outs is the same as an outfield error, which we pegged conservatively at .575. (This is almost surely too low, since the majority of difficult plays made in the outfield prevent extra-base hits.) That gives us a gap of about 25 range runs.

Adding together the run values of arms, range, and errors, we get a total of 41 runs. So it's certainly not, as Boz says, "silly" to think a single left fielder could cost his team 30-40 runs on defense. It seems pretty typical, in fact. And for a team like the Nationals without a single strikeout pitcher in the rotation, the stakes are even higher.

Whether Dunn is as bad as the very worst is a topic for another post.
When I do my own wins above replacement projections, I try to reduce the margin for error by averaging out three years worth of fielding data from at least a couple different systems. Going into this season, using three years of UZR and Rate2, I had Dunn as a 15.71 runs below average left fielder over the course of 150 games.

Regardless, it's a mistake to just dismiss out of hand the large number of runs that a poor fielder--even a poor left fielder--can cost you. And a run allowed in the field is worth exactly the same amount on the scoreboard as a run scored with the bat.
  • Update: The weekly UZR update is up at Fangraphs. With the usual caution about small sample size (UZR really isn't all that reliable on a week-by-week basis), but with a grain of salt his numbers are creeping up, as some of the commenters suggested. His UZR hasn't sunk at all from -7.5, meaning he's been average for the last week. His season-long projection (UZR/150) has risen to -28.


Anonymous said...

My problem isn't the OF stats, it's the 1st base stats. Those stats say that Dunn would cost the team -40 runs over a full season. I find it hard to believe that a 1st basemen can have that kind of adverse effect considering they get like 2-3 balls per game. Plus, My eye tells me that Dunn has only missed something like 3-5 balls that a 1B "might" get not 40 runs worth.

Steven said...

I'm not sure I follow. First basemen handle the ball more than anyone on the diamond other than catchers.

Also, as was correctly stated in one of the comments the other day, none of the advanced metrics show that Dunn is a -40 runs below average 1B anyway.

In the pre-season WAR post, using 3 years of UZR and Rate2, I had him at about 15 runs below average at 1B as well.

The only stat that comes close to that is the very small sample size UZR/150 for this season. Looking at that number isn't much different than looking at a guy who hits 2 homers on opening day and saying he's on pace for 320 HRs. It's fun barstool talk, but nothing more.

But Dunn's career UZR/150 at first is -16.9.

Of course, if the best fielder in the league is saving 7 runs above average at first (Casey Kotchman), and Dunn is costing 17, that's 24 runs.

Deez Nats said...

All I know is that Dunn was horrendous in LF. Outs became singles, singles became doubles, and doubles triples. Anecdotely, he seems to do less damage at 1B.

JayB said...
This comment has been removed by the author.
JayB said...

Errors can be throwing are double counting in some sense, but Dunn under Acta had free rein to lose focus in the field and did cost runs in bunches in LF.

Dunn should have been at first earlier because Johnson should have been traded earlier.....much earlier, like May 1st when Acta should have been fired.

Steven said...

There are throwing errors, "kills," and "holds." These are counted separately and not double-counted.

JayB said...

I'll have to take your word on that....the words you presented did not try to account for this fact. Cost of error would be different as well depending on fielding or throwing error.

If you accounted for this as you say I would like to know what the brake down of LF errors many fielding vs how many throwing.

Scott said...

Hey Steven,

I'm a new subscriber to the blog and really appreciate your take on things. I have to side with Boswell on this one though, as you defend the validity of the fielding metrics by conglomerating a whole bunch of fielding metrics!

In addition, you make a lot of assumptions about the accuracy of the various statistics you include in your analysis. I am highly skeptical that anyone can determine a "hold" by reading play-by-play data. There is simply no way to assess all the variables (runner speed, velocity of ball in play, time for fielder to track down ball, distance of throw) and determine if it is truly the throw that is "holding" the runner.

Another quibble is that your analysis does not account for the ability of the accompanying centerfielders. Surely a poor leftfielder will yield fewer runs if paired with a great center fielder, right? (Admittedly, this was NOT the case in the pre-Nyjer days.)

40 runs seems high to me, but then again, MY only evidence is the raised eyebrow I got when I read your post.

Maximus said...

Good stuff. Boz is the everyman's analyst. It is difficult to put your head around the fact that a player may cost you 40 runs a season in the field, but the stats, even if there is some margin of error, show that Dunn costs you a lot out there. At his salary, your really not getting a good value.

Anonymous said...

I'm sorry Steven, but I just can not see how the best fielding 1B would be worth 40 runs over a season over Dunn. That's just way too many. Sure the 1B handle the ball more, but I can't really recall a moment where Dunn dropped a throw. He's had some bobbles and range issues but not to the tune of 28 runs for a full season.

Steven said...

I think I said the gap between Dunn and Kotchman is more like 24 runs, not 40, but regardless I think we need to do better than just saying it doesn't "seem" possible. The analysts who are sitting down and charting out every single play in order to create the most objective possible tracking deserve more attention than that. The stats aren't perfect, but they're better than relying on intuition alone.

Seattle Steve said...

Ayyyymen, Steven. It truly irks me that people who haven't taken the time to dig into what goes into forming these advanced metrics can be so dismissive. It's not easy having the common wisdom that gets passed of as "analysis" in the major sports media get challenged. But the fact is, some really smart people are spending waaaaaay too much time looking at these numbers, and the plays behind them to just dismiss it out of hand.

And the proof is in the pudding. Look at Jarrod Washburn's ERA with Seattle (near the top in UZR) and with Detroit (near the middle). This is no fluke. The Nats are seeing the difference since the outfield defense was improved. It's not magic, folks.

JCA said...

In fairness to Boz, the number he was ridiculing was not the difference between the worst and best defensive outfielder and first baseman. Instead, he was criticizing the assertion that an outfielder or a first baseman could cost 30 - 40 runs more than an "average" fielder.

The sample size problem for Dunn at first is probably a big chunk of the reason he had extreme numbers earlier this year, and you picke dup it is starting to correct itself.

I found the links on this fangraphs post about Mitchel Lichtman and UZR to be a great read. Lichtman reminds everyone that all UZR tells you in small samples is his UZR score. It is when you get an adequate data set that the number becomes more powerful, and that is can take years for a UZR number to justify a conclusion about a player's true defensive ability.

DMan said...

Metrics aside, now that it appears that Dunn is installed at first base and can now concentrate solely on working on that skill set, do you think he can become an average fielding first baseman? IMO, he has reasonable ability in receiving balls for the 1B put-outs.

Steven said...

can he? yes. will he? I'll bet the under. big time.

DMan said...

Oops, keep forgetting one of the big criticisms of Dunn is his OTA (to borrow from the football lexicon) dedication.

e poc said...

You kind of proved Boz's point, though, right? The difference between the best and worst left fielders is about 41 runs. The difference between worst and average is likely to be 25 at the most. So the numbers back up Boz's assertion that it's next to impossible for a left fielder to be 30-40 runs worse than average on defense.

Steven said...

I don't think that's what he said. he said the fielding metrics are wildly inaccurate and dubious. There aren't any advanced metrics that find him at 30-40 runs below average in LF, so that's just a straw man based on his own lack of understanding. He didn't say what he thought was accurate. He just asserted "some" based on his gut.