#47 Spike (see all posts) 2010/05/03 (Mon) @ 03:26

Wait a second, wait a second, wait a second!!!

MGL said:

FWIW, I have Howard as #133 in my projection database for 2010… [SNIP] Here are some of the luminaries that are projected higher than Howard that would cause me to be committed as quickly as possible by ANYONE in the Phillies’ front office:

Elija Dukes

[SNIP]

PHELPS, JOSH
GERST, KENT
LAFOREST, PETE
JASO, JOHN”

While this is a cogent argument for why the Phillies are absolute buffoons for the Howard contract… isn’t this more proof that we have to do a validity check on the person who is supplying the argument?

Seriously??? Kent Gerst??? Gerst has a lifetime batting line of .251/.323/.339 in the minors. He’s 22, was cut in spring training and, to my knowledge, has been picked up no one. And this guy is worthy of being in the top 133 of a projection database?

The others, with the exception of Elijah Dukes, are almost as laughable as Gerst. And Dukes… well, how’s that been working out?

But, in the end, if a novice came around these parts with projections that had the above names in the top 132, my assumption is that you would all ridicule that person into oblivion. Since this is MGL, he gets a pass?

Seriously… at some point someone has to call shenanigans on at least Kent Gerst, right? I mean, maybe Pete LaForest, too… but I’ll defer to MGL’s expertise on projecting the major league stats of a 32 year old corner infielder/catcher based on his numbers in the Can-Am and Mexican Leagues.

Having just formalized the MLP system into a “next year” system for both major- and minor-league players, this sort of embarrassingly obvious flaw is the material of nightmares. Whenever a systemic approach is taken, without a human “reality check”, there’s always a chance that a confluence of factors will make something blow up like this, and Kent Gerst will come out ranked in the top 81 players in all baseball in 2010 (not clear from the excerpt from “Spike”, but – at the link above – MGL announced that Jeter was ranked #82, and Gerst was better than him).

What are the components of the “process” which could drive such a ludicrous conclusion as this Gerst projection, and what steps could be taken to “automatically” adjust for these components in the future? Let’s start with Ryan Howard, since – despite the heated debates, there’s not a huge variance in expectations for how he will perform in 2010.

• The Heater preseason projections came in at .265/.366/.563 (+14.0 runs above average 1b – 650 PA).
• PECOTA Weighted Mean: .271/.355/.553 (35.2 VORP – 588 PA)
• MGL: “I have Howard as #133 in my projection database for 2010. He is +20 (per 150) in batting runs, -5.25 in base running, +1.12 in turning the DP, and +.9 in defense, and -12.5 for a positional adjustment.”

==> This isn’t a comparison of systems, but this interpretation assumes ~23 runs/150 for the delta between “average” and “replacement” (a rough calculation for this is 30 PA/run), making the MGL equivalent to VORP about +30.5 (+20 [per 150] – 12.5 [position] + 23 [vs replacement]).
.506

From the above, it’s clear that while MGL’s projection of Howard’s batting stats are probably less than those used for Heater and PECOTA projections, all the systems are at least describing a guy who is 30-something runs better than a AAAA first baseman, given 700 PA.

This is where things get hinky, though…

One could go back in time and calculate the “standard error” on these offensive projections, and – despite the lack of specificity for things like injury impacts – could come up with a good distribution curve for each of the projection systems here. Given that the differences among systems is so relatively small (from the ever-optimistic Bill James Handbook projections at .278/.372/.575 to MGL’s system which is more pessimistic than even Marcel – at .264/.355/.543), Ryan Howard’s expected (mean) offensive production is one of the most easily projected stats in baseball. Any variance calculation involves many assumptions of things such as which players are comparable to him, and how much to weight their stats (i.e. how “similar” are they), whether they are similar-worse or similar-better than Howard, and whether Howard’s constant good health and recent fitness improvements will have any impact.

The factors which make the choice of assumptions for a Howard projection end up creating a fairly high variance within a projection for a player who has such similar cross-system projections. PECOTA’s percentiles give a sense of this, as the VORP by percentiles read:

90th – 64.0
80th – 55.0
70th – 48.6
60th – 41.8
50th – 24.1
40th – 21.6
30th – 17.4
20th – 15.3
10th – 12.0

The ins and outs of projecting Ryan Howard’s offense could easily take up an entire research piece, and many good articles have been posted around the net on this exact topic, including a good monetary impact discussion at Baseball Prospectus by Matt Swartz. But the bigger point here is that there is a reasonably high level of uncertainty in these projections, despite many factors helping to increase the ability to project his stats:

- “Sample Size” of over 3000 PA entering 2010.
- A huge percentage of his value coming from home runs, which have one of the highest rates of year-to-year correlation among batting stats.
- Limited portion of value coming from defense

[ed - moved the text above, as it was placed wrong in the original]

That brings the discussion back to Kent Gerst. One would think that a researcher with integrity would view a projection of Kent Gerst being better than Derek Jeter and Ryan Howard (in 2010) as a serious flaw in the process (hence the title). Gerst’s seasonal age in 2010 will be 22. He was cut at the end of March, and “Spike” above may just not have Googled enough, because he was indeed picked up:

The Windy City ThunderBolts of the Frontier League have announced the signing of former 8th round draft pick by the Chicago White Sox, Kent Gerst. Gerst is a huge addition to the outfield, as he will likely handle the centerfield duties for much of the 2010 season. (wcthunderbolts.com)

MGL goes on to explain that he used “MLEs” to project minor-league players. Without transparancy to his system, the MLP/Heater MLE values and Dan Szymborski’s from Baseball Think Factory (for anyone interested in the methodologies for calculating MLE’s, Dan ’s article from a few years ago is a good starting point). Within the framework of some common assumptions, for guys like Gerst, there’s still some variance, as shown by the rate stats for Szymborski’s zMLE’s and the MLP System MLE and BP’s Eq stats:

• zMLE: .171/.226/.251 (2009 stats)
• MLP System MLE: .210/.277/.328
• BP Eq Stats: .209/.280/.328

Assuming the best for Gerst here, both offensively and defensively (he played some side outfield, so it’s unclear that he’d be able to stay in center field defensively), he would still be worth about -26 runs compared to an average center fielder, given his 2009 performance (using the Heater MLE) and 650 PA. Adjusting that for 650 PA of replacement player, he’d have been worth almost 5 runs less than a replacement centerfielder in 2009 (given 650 PA).

Looking forward to 2010, PECOTA forecasts some negative growth, with Gerst dropping down to -10 VORP in 313 PA. For the sake of argument, however, the assumption that a 22-year-old ballplayer will improve will be used, and his -5 runs per 650 PA will be augmented to become 0 runs per 650 PA… indicating that Gerst would be an almost perfectly typical “replacement level” centerfielder.

The first conclusion here is that while some optimistic assumptions have definitely been used, the fact that Gerst is just 22 and already profiles as “replacement level” suggests that cutting him might not have been the best move in the world, barring the existence of serious negative “intangibles” (see: Bradley, Milton).

Anyway, deducing the fielding-plus-baserunning values which MGL’s system must be attaching to Gerst, it comes out to something greater than +25 runs. This deduction is made by the conclusion that Howard is worth +2.5 WAR in that projection system, and Gerst is better than Howard.

Gerst is not a top-tier speed guy. With the top EqBRR values in the majors last year being around +3 runs, and fewer than 30 players reaching +1 EqBRR, there’s little reason to assume that Gerst is better than +1 baserunning runs.

That leaves well upwards of +20 fielding runs, even with all the assumptions above being made to increase Gerst’s value, and hence reduce the implicit number of fielding runs required to make him worth more than +2.5 WAR in 2010.

And this, finally, is the very likely culprit in this nonsensical ranking of Kent Gerst. MGL has been well-publicized as stating (rightly) that fielding data really doesn’t gain statistical significance for about 3 full seasons of data. Gerst has just 219 PA of full-season experience. He’s had 79 chances in the outfield.

Two things conspire to make the simple addition of “fielding runs” to “offensive runs” and comparing player-to-player an untenable way of rating these players:

1. +20 Fielding runs is essentially a boundary condition. While most “expected value” results come about because of the possibility of both upside and downside, there’s essentially no “upside” with an expectation of +20 Fielding runs (per 150). Sure, sometimes a Franklin Gutierrez comes along, and posts +31 UZR (+26/150), but that sort of performance is several standard deviations from the norm, to the extent that it has to be considered virtually impossible; and the “downside” variance is very significant, especially when any injury can dramatically influence the range of an elite fielder.

2. 79 chances? Is this a joke? Sure, some balls which he didn’t catch are also included as data points, but for someone who is projected to have a huge defensive impact, these aren’t very likely. Whatever happened to a reasonable rate of regression? The state of minor-league fielding data is still far behind, and while some try valiantly to measure it, they end up with conclusions such as Colby Rasmus and Austin Jackson being two of the worst fielding center fielders in all baseball when they were in A ball (see comment to this post). Regardless of the state of the data, regressing any possible data in such a tiny sample size to minor-league norms for fielding would negate any actual stats as being nothing more than “noise” with nowhere near enough “signal” to draw any conclusions. Contrast this with 3000+ in sample size for Howard’s batting stats.

In summary, the reason Kent Gerst came out ahead of star players like Jeter and Howard is almost certainly due to some sort of embarrassing mistake in the system, and though a first reaction of defensiveness is normal, one has to assume that the quest for accuracy will win out and the error(s) will be rooted out of MGL’s system. But it is an interesting thought exercise to approach the question with the assumption that a 22-year-old Indy League player is actually worth more (in 2010 wins) than Ryan Howard (or Derek Jeter), and try to figure out what assumptions that conclusion would require.

I have just summed up most baseball fans’ reaction to the recently-released projections. How did you like my accent?

The above, however, is not what PECOTA is trying to tell you.

Colin Wyers sums up the limitations in a comment at Baseball Prospectus:

Even if we are correctly predicting everything – everybody’s raw hitting and pitching stats, everyone’s playing time, etc. – correctly, there is still some random variation.

How much? I mean, a lot. The standard deviation of win percentage over 162 games simply due to random variation (or “luck,” if you prefer) is a little over six games. Events within one standard deviation occur 68% of the time.

When looking at the standings, your gut reaction is to criticize the Rays being projected to win the division with the World Series champion Yankees finishing third.

I made the following “chart” in Excel, highlighting each AL East team’s win range within one standard deviation above and below the projection. Hopefully you can match the color with its representative team.

Notice how many of the win ranges are shared by all three teams: 90-99. PECOTA is saying that it believes the Rays to be better than the Yankees, but the Yankees could still win the division a decent percentage of the time due to, among other factors, random variation.

Angels fans are irate that PECOTA predicts the team to win a mere 76 games. If we go to two standard deviations, between which data falls 95% of the time, the win range spans from 64-88.

“Great,” you say. “Even I could predict the Angels to win between 64-88 games.”

It seems trivial, but that is because people are expecting PECOTA to make grand claims when it’s really not going to do that. If you’re expecting PECOTA to mimic Nostradamus, you are always going to close your Internet browser disappointed.

PECOTA projects playing time as well when calculating those standings. For instance, Jimmy Rollins has averaged 717 PA per season since he started playing regularly nine years ago. PECOTA projects him to attain only 647 PA in 2010. That’s because PECOTA is projecting at the 50th percentile, or the mean. Rollins is projected to be worth 4 WARP in those 647 PA. If he gets those extra 60 PA, he may add another 0.4 WARP.

Rollins has stepped to the plate very frequently over the years, but still, wouldn’t you bet your money on him attaining 647 or more PA rather than 717 or more? That’s what PECOTA is trying to tell you: what is most likely to happen at the 50th percentile (mean). It’s nice when it hits right on the nose as it did with the White Sox in 2007, but it’s much less likely to get it exactly right than wrong. In fact, you could probably go back to the individual ‘07 White Sox projections and point out some errors.

Don’t get bent out of shape over PECOTA projections. No projection machine will ever correctly predict baseball players and team performances with any acute accuracy simply due to random variation. PECOTA won’t predict the Rays to win the AL East in 2010 every time for the same reason why you won’t (hopefully) bet on Roger Federer winning every major this year. Some days, Roger will be off; other days, his opponent will be on. Try that exercise 162 times for two teams separated by 3 wins.

As Vegas Watch points out, PECOTA is the best in the business at doing what it does. Anyone who thinks PECOTA is useless or consistently wrong should publicly display their own projected standings (with a transparent and detailed methodology) and put it up against PECOTA at the end of the season. I’ll put my money on PECOTA all day, every day.