[Pitchers] do not go gently into that good night.

Well I couldn't find a retirement announcement database online, but I had a couple of hours to kill on Amtrak, so I hacked something together. I thought my findings might be useful to someone, so I'm posting the individual cases in a spreadsheet here.

For seasons 2005-2009, I found all pitcher who threw no MLB innings, despite being projected for 60+ IP by my system. My system does not give 60 IP projections lightly (see previous post), so these cases are fairly rare. In fact, there were only 43 cases (something like 2% of all pitcher seasons), all of which I researched manually during my train ride.

Digressing for a minute, the fact that only 2% of veteran pitchers (I never project rookies to throw 60 innings) each year don't play in the majors when they are projected to pitch 60 innings is stunning. Remember, 60 IP is the average innings per MLB pitcher each year. Even with injuries, retirement, and ineffectiveness, pitchers who are projected to command average MLB innings do managed to book some MLB time 98% of the time.

Of the 43 candidates, 15 met the requirements of "voluntary retirement" that I was looking for. I was looking for pitchers who announced their retirement during the off-season or during spring training. Being cut in spring training & retiring counts. Retiring as a free agent also counts, as long as the pitcher did not appear in the minors, on the DL, or ever pitched in organized baseball again. Also two pitchers died. Lastly, I included Darrell Rasner on this list. He was sold to Japan during the offseason in a deal that he approved. He clearly could have pitched in the majors, but instead chose to go to Japan. Similarly, Kaz Ishii returned to Japan after a few years in the states. Others played in Japan after being cut in spring training by MLB teams. I did not count those guys.

I made notes on just about all of the 43 pitchers, so feel free to come to your own classifications. Also I noted whether the pitcher missed the entire (0 IP) season to injury, or whether he pitched in the minors that year. That data could be useful also.

Of course, I am not considering any voluntary retirements for pitchers projected to throw less than 60 IP. But I think that those retirements are less worthy of special consideration. It's not clear that Randy Johnson or John Smoltz could make a major league roster in 2010. Maybe they could, but it's less than clear, and I don't think anyone would project them to be anything more than a 5th starters. However Mike Mussina (and also Greg Maddux) could have absolutely commanded a starting rotation spot in 2009 if they chose to keep playing.

I'm re-training my IP models and should have new projections up soon.

As a quick look-ahead, I trained on the new data (with 15 voluntary retirements excluded from training) using the old models for IP_Start and IP_Relief as features. Relief innings projections don't change, while the new data suggests that IP_Start should be projected to be 0-2 IP higher for all pitchers. Although the later will not actually happen in my fully retrained model.

When I fully retrain, I expect to see the high-IP starters get more innings, while innings for the rest remain the same. Also, the age-based penalty for IP_Start should decrease, as voluntary retirements being excluded here mostly pertain to older pitchers

2010 IP projections

Based on the simple ML system that I wrote about in my previous post, here are my 2010 innings pitched projections. I offer predictions for IP (broken down by starter innings and reliever innings) for all pitchers active in 2009, as well as for those who missed 2009, but had over 120 IP in 2008 (ie Ben Sheets, Mike Mussina, etc). The pitchers are ordered by 2009 IP.

I also offer my predictions from an earlier system that does not account for pitchers’ value stats (wins, VORP, saves, etc). I include the difference between value-drive and non value-driven projections in the last row. These differences range between +30 IP to -30 IP.  The IP_Start model rewards both 2009 value and previous value, while the IP_Relief model is more heavily weighted toward 2009 value. Roy Halladay and Johan Santana gain the most among high-impact starters, while Mariano Rivera and Joe Nathan gain most among high-impact relievers. More on that later.

It would be in poor form for me to include PECOTA or CHONE predictions next to mine, so let me share a few averages with you instead. In this context, rookies are pitchers who had at least 1/3 IP in 2009, but less than 50 IP in their careers. Projections for rookies with no MLB experience are not included. My system does not use minor league stats or scouting reports, so all such rookies will project to about 20-30 IP, depending on age.

	Actual  2009	My system (2010)	PECOTA (2010)	CHONE (2010)
Mean IP	65.1 IP	58.2 IP	99.9 IP	91.2 IP
Mean IP non-rookies	78.2 IP	66.4 IP	103.6 IP	95.1 IP
Mean IP rookies	14.8 IP	26.4 IP	86.0 IP	76.4 IP

My system expects an average pitcher to regress by 7 IP from 2009. Since total innings must add up, the roughly 500 * 7 = 3,500 missing IP will be made up for by (yet unknown) pitchers without any major league experience. That’s about 100 IP per team.

In my system, veteran pitchers will regress by about 12 IP from 2009 on average (including pitchers who retire; more on that later). Rookies who’ve had a cup of coffee in the majors  will expect to pitch almost 12 IP more in 2010 than in 2009. All of this is pretty consistent with the averages that I outlined in the previous article.

However CHONE and PECOTA predictions are not consistent with these averages. In either system, the average pitcher will increase his innings pitched from 2009 by over 30 IP.  The average rookie will increase his innings pitched by over 60 IP! The average major league rookie is projected to throw more innings than Mariano Rivera.

I don’t think that any of this is realistic. CHONE and PECOTA allocate over 100% of 2010 innings to veterans, then allocate a further 20% of 2010 innings to rookies, and none of this accounts for pitchers making their major league debuts this year (while my system leaves 100 IP per team for those pitchers).

Some may argue that PECOTA and CHONE make “if he makes the majors” projections. A lot of these innings will end up getting pitched in the minors, and major league innings can be adjusted on a team-by-team basis. I think BP has a manual process where pitchers are selected based on likely playing time, and PECOTA IP projections are adjusted accordingly.

However I’m not sure that such a process is necessary, nor do I think it’s optimal for projecting team pitching totals. As I showed in my last posts, there are pitchers every year that come out of nowhere to pitch significant innings. Also the top prospects, as a group, collectively pitch fewer innings than CHONE or PECOTA would lead you to believe. Lastly, there was a nice article on BP recently (by Tommy Bennett, behind the pay wall) showing that a team’s 5^th and 6^th starters pitch comparatively similar innings on most teams. Therefore picking which players will be in the rotation, or who will be on the 25-man roster, is both futile and also counterproductive for projecting individual IP.

It would be better to compute an independent set of IP projections that respect the recent averages, and then (possibly) make some small team-based adjustments by hand or automatically. It’s more realistic to say that the Yankees will use 20 pitchers in 2010 whose collective IP is slightly less than the Yankees’ overall 2010 IP, rather than listing their top 10 or 13 pitchers, whose collective IP adds up to or exceeds the Yankees’ overall IP.  (FWIW, the Yankees used 24 pitchers in 2009.) Now using projected ERA or another value rate stat and a realistic IP projections, it should be possible to approximate the Yankees’ overall projected pitching value in 2010 in a robust manner.

Hopefully I’ll have time to complete all (most?) of those steps before the season gets under way.

Although my system is much better than CHONE or PECOTA at generating IP estimates for rookies, and somewhat better at projecting veterans, it is still far from ideal. Most notably, I think my (non)handling of retired pitchers hurts my system’s ability to project veteran starters’ IP rationally.

I think it’s important to train a system with examples where veterans don’t come back strong after a bad or injury-plagued year, and also to include examples of sudden falls to 0 IP after decent seasons. Both these things do happen to pitchers and the model need to take that into account. However, I think my system is overdoing the downward regression bit for starting pitchers. The only pitcher who threw 130 IP in 2009 who is expected to increase his total in 2010 is Johan Santana (who is expected to move from 166.7 IP to 174.0 IP).  A typical starter who threw 180 IP in 2009 with solid results (Roy Oswalt or Joe Saunders) is expected to throw 30 IP less in 2010. This sounds a bit harsh to me.

The cases of voluntary retirement for guys throwing over 100 IP in the previous season are few, but they might be screwing the results for all starting pitchers in my model. Greg Maddux and Mike Mussina were projected for 140 IP and 180 IP respectively in 2009 by my system. Both retired and threw 0 IP. In a model designed to minimized root mean square error, a few such examples can make a significant impact. Both Mussina and Maddux retired well before the 2009 regular season. My model should have been privy to that information, and ascribed their severe downturn in IP to retirement, and not so much to other factors.

Ben Sheets and Jeff Francis were also projected to throw a bunch of innings in 2009, but missed all of last year due to injury. Their cases should absolutely be in included in the model. However before I can build a decent injury-based model for IP, I need to account properly for known preseason retirement. Now where can I get that damn retirement data…

Excessive exuberance for young pitchers? (III)

Now that I have showed that my "simple linear weights model" predicts IP better than CHONE and PECOTA, I should probably explain how my model works. However, I'm not sure how much anyone who's reading this cares about what features I used or how I avoided over-training the model. So I'll keep this brief, share a few links, and promise to have my 2010 IP projections up here soon. Maybe as early as tomorrow.

I trained my model in pieces. Most importantly, I trained separate models for IP_Start and IP_Relief. Here is a simple graph of what I did:



My basic inputs were:

• "IP stats" (such as IP, IP_Start, IP_Relief and also DNP (did not play), ROOKIE, etc) for the previous four years
• "Value stats" (such as wins, losses, holds, VORP, SNWP (support neutral winning percentage for starters), WXR (wins added in relief), etc

To get the final model, I first trained basic models for IP_total, IP_Start and IP_relief purely based on previous playing time. Then I trained the final IP_Relief model based on the basic projections and also the value stats. The final IP_Start model uses this same IP_Relief model, as well as value stats. This allows the final projection to be a simple sum of the IP_Relief and IP_Start models.

Each of the five trained models can been seen in a long text file here. Most of the features are self-explanatory, although a few are not. I'd be happy to explain what they mean, if anyone wants to know.

Note that "_2008" features simply mean features for the previous year. I named all the features as though I were training on 2009 pitcher seasons, although I actually use 2005-2009 pitcher seasons for training. Also "_average" features are averages for the previous three years not including the last one (ie "_2005", "_2006" and "_2007"). The "_ip_average" features (the few there are) are four-year averages, weighed by seasonal IP. So they are a sort of "how did he do when he actually played" features. They are very optimistic estimates, in other words. Most of the features used are "_2008" features, which shows just how much projections are dependent on very recent performance.

Again, I won't spend much time on these models. Each model reports a correlation with the data that it is trying to predict (based on 5x cross validation). The correlation is the "r" in "r^2" that you statistically-minded people know and love. My most basic IP_total projection system gets a 0.71 correlation with the data (using previous year's IP only, and not included here), while the final system claims a 0.76 correlation. That doesn't sound like much, although that's a 10% improvement in r^2.

More importantly, my final projections take account of a broader set of features than the more basic projections. Statistically speaking, there is not much gain from looking at value stats when projecting IP. We know that a pitcher who throws 200 IP and is very effective is more likely to throw 200 IP again than a guy with the same 200 IP who was at replacement level. However, the difference in ability is usually reflected in the previous usage anyway (replacement level pitchers don't often accumulate 200 IP), so the information gain is not large.

However I would rather use this information, even if the gain is small. I want my system to take account of the fact that Mariano Rivera provides lots of value, and thus project him differently than Brad Lidge, who had a similar number of innings pitched in relief. Even if performance and playing time is highly correlated, I would rather use more factors, as long as their relationship makes sense to me, and as long as I am not using too many.

People in AI often say that feature selection is an art, as well as science. If there is no information gain in a feature, my system won't use it. However, most of the relationships I use in my models are not statistically significant. Which features I keep and which ones I don't keep is a matter of discretion. Generally, I want to use features that make intuitive sense, and which comprise both "_2008" information, and information from previous years. I could train a model with almost the same accuracy using only "_2008" features, but it would consider Brandon Webb and Ben Sheets as old rookies going into 2010. We know that's not the case, so I'd rather have a model that as least tries to use all four years' worth of data.

As it turns out, older data is useful to predict starters' innings, but is almost useless for predicting future relief innings. Here is the final IP_Relief model:

IP_Relief_2009 = 

0.4164 * L_2008 

+ 0.3454 * BS_2008 

+ 0.3444 * VORP_2008 

+ 0.2825 * Hold_2008 

- 4.9235 * SNVAR_2008 

+ 1.9615 * WXR_2008 

- 3.3587 * bin_change_LG_2008 

+ 0.1924 * SNWP_R_2008 

- 0.3574 * W_average 

+ 0.4407 * L_average 

+ 0.078 * VORP_average 

- 0.9571 * SNVAR_average 

+ 0.8097 * LG_average 

+ 0.1252 * proj_IP_v26 

- 0.1387 * proj_IP_Start_v26 

+ 0.4968 * proj_IP_Relief_v26 

+ 5.1194

You don't need to understand what all these features mean to see that next year's relief innings are heavily dependent on WHYDFML. Total value added (with value added for starting pitching taken out) is responsible for 1/2 of relief innings pitched prediction, while only 1/2 is dependent on the simple IP_Relief model that accounts for previous years' innings pitched. Therefore an effective reliever who pitched 40 innings in 2009 might be projected for 60 IP in 2010, while a reliever who throws 80 IP every year might be projected for less than 60 IP next year, if his last year wasn't so good. Having been an effective reliever in 2007 means nothing, if your 2008 sucked.

I'm not sure why SV (saves) did not make the cut for features used to predict IP_Relief, while Holds, BS (blown saves) and WXR (wins added in relief) proved to be useful indicators. Maybe this goes to show that saves statistics get abused too much nowadays. A setup man with twice the holds and double the WXR as another setup man is most certainly a more valuable setup man, but a pitcher with 60 saves is not inherently more valuable than a pitcher with 30 saves.

The IP_Start also has an interesting caveat:

IP_Start_2009 = 

0.8554 * W_2008 

+ 0.877 * VORP_2008 

- 16.6386 * SNVAR_2008 

+ 2.8454 * SNWP_R_2008 

- 0.606 * L_average 

+ 0.5458 * VORP_average 

- 2.2624 * SNVAR_average 

+ 0.2685 * proj_IP_v26 

+ 0.6921 * proj_IP_Start_v26 

- 0.761 * proj_IP_Relief_v26_w_value 

+ 6.1707

Here "SNWP_R" is a BP feature (Support Neutral Winning Percentage) that I turned into an "over replacement" stat. This is BP's stat for measuring starter quality on a 0.0-1.0 scale where about 0.44 is replacement level (and 0.6 is really good). I smoothed values for low-IP guys and count the points above 0.44 (multiplied by 100). 

What's interesting is that SNVAR (wins added as a starter, also from BP) is negative. Positive values for VORP and negative ones for SNVAR mostly neutralize each other for starters, while SNWP_R is always positive. Therefore a starter who is really good in 160 innings will get the same credit as a starter who is equally good in 220 innings (same SNWP but higher SNVAR).

The basic proj_IP_Start_v26 (IP & rookie stats only) IP_Start model is essentially a regression of last year's IP_Start (with extra credit for previous years' IP), so it's interesting that the bonus for a good 160 IP pitcher is the same as that for a good 220 IP pitcher (although the later is hurt more by regression in the simple model). The model does not a have a feature to say "this guy has pitched 220 innings every year for five years, so I should give him credit for that skill," but still, it seems to suggest that such pitchers are more lucky than skillful. As Brendan Webb showed last year, being a workhorse for years does not preclude a regression to the mean. It's easy to say that a guy is special in retrospect, but projecting into the future, there is no reason to give any of today's pitchers a 50th precentile projection of 220 IP.

I'd be happy to discuss more details about these models if anyone is interested. Otherwise, I'll proceed to 2010 projections and thoughts for further improvements. Including a better understanding of how injuries affect IP prediction.

Also, I need to repeat these steps (and some other steps) for ERA prediction, and VORP prediction (although hopefully ERA and IP will interact well enough to produce VORP projections directly). 

Excessive exuberance for young pitchers? (II)

In my last post, I claimed that projection systems tend to over-estimate playing time for young pitchers. I went on to claim that a system that ignored minor league stats and treated all rookies equally would give better overall projections for pitchers’ playing time. As it happens, I’ve built such a system.

As I have written about before, I’ve been working on a pitcher season projection system for some time. I started building my projection system with data from Baseball Prospectus, which does not include comprehensive minor league stats. I’ve never bothered to acquire minor league stats elsewhere, so in my system, all I know about a rookie is that he is rookie, and nothing else. Actually, I know other things (skills ratings, Baseball America ratings, fastball velocity), but here I will demonstrate a simple projection system, that only uses a pitcher’s basic MLB stats, rookie status, and age.

I will explain my system later. For now, let’s just say it’s a linear-weights system based on the past four years’ stats such as:

• IP_Start, IP_Relief, rookie status & missed seasons
• Wins, losses, saves, hold, VORP (total runs saved), WXR (relief pitching value in expected wins), etc

I trained my system on pitcher seasons from 2005-2009 that either:

• Recorded at least 1/3 IP in the majors that season
• Recorded at least 1/3 IP in the previous season

Therefore, my system takes account of the fact that, each year, 20% of pitchers do not return to the big show.

Perhaps I should remove the voluntary retirement seasons. However most pitchers retire when they can no longer pitch in the majors, so the current system is reasonable. In any case, it is better than not accounting for the fact that some pitchers who had decent seasons go directly to zero the next year.

After building my system, I commit the sin of training and testing on the same data. However, I could not leave out 2009 for testing only. I am using very little data anyway (only five years’ worth). I could give you the testing data on clean 2004 data, and later I will, but I doubt you’d. Just trust me that the system was built in a way that reduces over-training and keeps things very simple. If you are interested in the details, please read my next post.

In any case, I compared the predictions of my system for 2009 IP with last year’s preseason projections from CHONE and PECOTA. I tested on all pitchers where my system, CHONE and PECOTA all provided predictions. This consisted of all pitchers who had 2008 or 2009 major league IP, sans about 40 pitchers that were missing in either PECOTA or CHONE.

None of the omissions were notable pitchers. Although I do think that it’s notable that projection systems that project huge numbers of minor leaguers (mostly with unrealistic stats) still manage to miss some pitchers who make it to the majors. I think this goes to show the point I made in the last post: we really don’t know which minor leaguers will get playing time, so we might as well just play the percentages. But I digress...

It would be in bad form to post PECOTA and CHONE projections here, since I do not own this data. Therefore, I will just summarize the results, and offer a few examples.

For the full set of 2009 pitchers, my average projection error is significantly lower than either PECOTA or CHONE. Also, my IP projection beat PECOTA head-to-head 63.8% of the time, and it beats CHONE 66.3% of the time. Most surprisingly, my projection is best out of all three 54.8% of the time!

	My system	PECOTA	CHONE
IP error STDev	43.6 IP	55.5 IP	61.6 IP
Best projection	54.8%	23.3%	22.2%
Mean error (IP)	-0.4 IP	25.8 IP	32.2 IP
Mean IP	54.0 IP	80.1 IP	86.6 IP

The last two rows here show just how much PECOTA and CHONE over-estimate playing time for an average pitcher. If we include O IP seasons for pitchers who played in 2008, the average pitcher in 2009 threw 54.4 innings. And yet, PECOTA projected the average pitchers for 80 innings, and CHONE for even more. These are not reasonable baselines (even if we removed the empty seasons).

You may argue that 0 IP is never a good projection for a major league pitcher. That is true. However, Ben Sheets, Tom Glavine and Kei Igawa all looked to have reasonable shots to pitch in the majors in 2009, and none of them did. I think my system does deserve credit for projecting lower totals for guys who don’t pitch at all.

Actually, my system still easily beats CHONE and PECOTA if we remove all 0 IP season. To make closer, I remove all seasons with <20 IP. With that, the average pitcher IP rises to 87.6, and all of the systems are equally effective:

	My system	PECOTA	CHONE
IP error STDev	45.9 IP	43.3 IP	47.0 IP
Best projection	34.1%	32.3%	33.8%
Mean error (IP)	-19.7 IP	2.9 IP	7.1 IP
Mean IP	67.9 IP	90.5 IP	94.7 IP

Ok, so my system does better because I project fewer inning for pitchers who retire, right? Not quite. Pitchers that both PECOTA and CHONE projected to throw over 100IP, but who threw less than 20 IP include:

• Ian Kennedy
• Robert Mosebach
• Mitch Atkins
• Andrew Carpenter
• Zach Johnson
• Brandon Webb

Chances are, you know who 1.5 of those pitchers are, and only if you're a golf fan. No projection system saw Brandon Webb’s season-ending injury. However, what were PECOTA and CHONE doing projecting over 100 IP for the misfits?

As I wrote in my last post, I think these systems over-value minor league stats, and give unrealistically high projections to young pitchers, and especially to rookies.

Consider how the projection systems perform if we keep 0 IP seasons, but remove all seasons pitched by rookies (less than 50 IP of prior MLB experience):

	My system	PECOTA	CHONE
IP error STDev	45.2 IP	52.0 IP	59.1 IP
Best projection	48.3%	25.7%	26.3%
Mean error (IP)	3.0 IP	18.0 IP	25.1 IP
Mean IP	67.8 IP	82.7 IP	89.8 IP

As you can see, my system is still best (since CHONE and PECOTA over-estimate non-rookie pitcher IP also), but not nearly by as much as with rookies included.

Sure, these systems look great when they nail their projections for Jeff Niemann, David Price, Tommy Hanson, Brad Bergesen and Andrew Bailey. However they miss more often than they hit. If you say that Jeff Niemann and Tommy Hanson are better prospects than Ian Kennedy and Clay Bucholtz, it is precisely because they had better 2009s that you think so. Kennedy and Bucholtz were also highly rated prospects a couple years ago, and if they succeeded by now, we would have said “but of course I saw that coming.” And perhaps you would be right. However, as PECOTA and CHONE show, on average, these prospects struggle more than we think that they should.

Ironically, good projection systems that use minor league stats get crushed by a simple system that does not use minor league stats for precisely that reason. I’m not saying that minor league stats have no use. But we should not put too much stock in great minor league stats, since teams clearly don’t.