Firemen are soooo predictable! (addendum to 'Fastballs for Dinner')

Silly me. I was comparing starters to all pitchers. We all know that there are three kinds of pitchers: starters, relievers and Joba Chamberlain.

I isolated the starters with a simple 100 IP cutoff. I can similarly isolate the relievers by requiring no more than 6 starting innings pitched. This captures 86% of the relief innings, with less than 1% contamination by starter innings.

Graphs for relievers are below:

There is not a whole lot of difference between the known relievers and the "all pitchers" figures. However I can't help note just how similar all lefty relievers' repertoires really are. If this is evolution, I hope evolution doesn't happen to the general pitcher population.

Then again, the data I'm using does not take account of two-seam fastballs or of sinking fastballs. But still, all of the crafty lefties I can think of are starting pitchers. As I mentioned in an earlier post, there has been a steady increase of lefty starters in the past decade. So maybe clubs are doing a good job of getting their best lefties into the rotation. The lefties who throw smoke but can't learn a pitch other than a slider end up in the bullpen.

How do you know if he'll be unhittable?

After much delay, I'm ready to show my best effort for predicting strikeout rate from numbers not related to a pitcher's performance.

Say you've got a pitcher. College guy, minor leaguer, major leaguer, etc. You would like to project what his future MLB strikeout rate should be (granted that he makes it that far). Can we predict this number from scouting the guy (i.e. taking a look at what pitches he already throws, and projecting how his repertoire may evolve)? Yes, we can. With around a 0.6 correlation to actual performance. I showed my original model for this almost 3 months ago.

Since then, I have looked into a number of additional factors and adjustments. Also I can answer a few obvious questions raised by my original work. Most importantly, I am going to break down the model, rather than just drop a formula on my blog.

Starters & Relievers

I think it's important to know how well we can predict strikeout rate for starters, relievers, and for pitchers in general. So I will show three models.

I consider anyone a starter if he pitches 100+ innings. I consider a reliever anyone who throws less than 6 innings as a starting pitcher. This simple reliever classification captures 86% of the relief innings (with less than 1% of starter innings). The simple starter classification captures 82% of starter innings (with less than 10% of the reliever innings). Also, we avoid (almost all) pitchers who spend substantial time both in the rotation and in the bullpen.

M5 Rules in WEKA

I wish I could just show you a few multiple regression models side by side. However, the algorithm I use is slightly more complicated. Although only slightly.

I train my models with the "M5 Rules" algorithm in WEKA, the open source machine learning platform. This algorithm is basically a souped up multiple regression model. Given a bunch of features (30 or so in my case), the algorithm will build a linear model to predict a single value. With a few caveats:

1. The algorithm can split the data set along a linear rule (example: FB velocity > 90 mph).
2. The algorithm tries to keep the weights of features as low as possible. This also means that features that are not meaningful are eliminated entirely from the model.

If I give the algorithm 30 features, rather than getting back a single rule with 30 weights, I might get 2 or 3 linear rules, each with 5-10 weighted features.

In evaluating the accuracy of my model, I look at the correlation between the outputs of the model and the observed strikeout rates. However, the correlation values that I get are from 5x cross validation. This means that WEKA actually builds 5 models, each time leaving out 1/5 of the data to use for testing. The correlation figures are for the testing data of the 5 models. Therefore, I am never training and testing on the same data. However, when breaking down models, I always use a model trained on the entire data set.

The Three Models

As I explained above, each of my models (for starters, for relievers, and for all pitchers) will have 2-3 rules each. Thankfully, the models split along the same couple of features (FB velocity and starter/reliever classification), so I will be able to compare rules across models fairly easily. First, let me summarize the models.

model:	# of rules:	correlation with SO9:	average IP:	% lefties:
overall	3	0.5675	69.0	27.5
starters	2	0.6205	171.6	27.0
relievers	2	0.4456	35.9	28.1

You can see the models in a text document here. Also I have a grid of the features by model here. I have removed meaningless features in the spreadsheet. This should make it easier to see the significant features that are left over.

Let's quickly list the possible features:

1. "bio data:" handedness, height, weight, age
2. for each pitch [FB=fastball, SL=slider, CT=cutter, CB=curve, CH=change, SF=splitter, KN=knuckler]:
   
   • % thrown
   • velocity
   • whether the pitch is part of offerings
   • whether the pitch is part of depth
3. repertoire depth and repertoire offerings for the pitcher (an explanation with examples is here)
4. league (LG)
   
   • AL = +1
   • NL = -1
5. IP Start


• only for the 'all pitchers' model

Using IP in Strikeout Prediction Models

The concept of using IP in a model to predict strikeout rates seems contrary to my aim of predicting strikeout rates without using performance-based information. It certainly is. Playing time is the simplest measure of performance. Better pitchers throw more innings.

And yet, adding IP to the feature set above does not help us much in predicting strikeout rates. The 'all pitchers' model (which uses IP Start) has 0.5675 correlation with observed strikeout rates. If we don't allow it to use IP Start, correlation drops to 0.5491. The model stays just as predictive. However for correctness, I don't use any IP features in the starters and relievers models (beyond classification of pitchers into these models). I use IP for the 'all pitchers' model in order to have a rule separating starters and relievers. This makes later analysis a lot simpler.

The fact that IP is does not help us predict strikeout rates (if we already have pitch type data) is notable. This is not the case for the other pitcher rates that we might care about!

Let's restrict ourselves to starting pitchers. Here is what happens if we train models for various pitcher rates with and without IP as an input:

rate:	correlation with IP:	correlation without IP:	drop off:
SO9	0.618	0.620	-0.002
QERA	0.589	0.474	0.115
SO/BB	0.517	0.400	0.117
BBr	0.413	0.307	0.106
LD%	0.383	0.377	0.006
GB/FB	0.361	0.280	0.061
GB%	0.328	0.301	0.027
FB%	0.313	0.287	0.026
HRr	0.332	0.257	0.075
BABIP	0.221	0.091	0.130

As you can see, having access to IP data does not matter for predicting starters' strikeout rates. Given the same pitch distribution, a 140 inning fourth starter will tend to have the same strikeout rate as a 220 inning first starter. However this is not true for many of the other rates.

Pitch distributions & bio data does explain a significant portion of many of these rates. However the rates that we might care most about (QERA, BBr, SO/BB, GB/FB, BABIP) are dependent, do a large degree, on IP. I don't know whether this means that I am missing some method of getting more out of the non-performance related data, or that good pitchers have low walk rates and keep their BABIP down in ways that can't be measured using aggregated pitch data.

Once I am able to look at individual, rather than aggregated, pitch data from Pitch F/X, perhaps I'll have a better idea.

Fastball Velocity

As I've written before, average fastball velocity is the most predictive single feature for strikeout rates. This is true for all groups of pitchers.

Fastball velocity does not have a linear relationship with strikeout rates. Rather, as you can see from the graph here, differences in velocity matter a lot more on the high end (93-96 mph) then they do on the low end (86-89 mph). Therefore all three models split the data along high end/low end fastball velocities, thus creating separate rules for "hard throwing" and for "soft throwing" pitchers. This piecewise linear approach is a much cleaner way of handling nonlinearity than trying to fit a polynomial to the relationship between fastball velocity and strikeout rates.

Here are the weights for fastball velocity in the SO9 models. Remember that SO9 is measured in strikeouts, while pitch velocities are measured in mph. So a weight of +0.50 means that an increase in 1 mph on the fastball will result in a SO9 prediction that is 0.5 strikeouts higher.

model:	hard throwers:	soft throwers:
starters	+0.519	+0.316
relievers	+0.586	+0.351

The harder that a pitcher throws, the more he has to gain (strikeout-wise) by throwing even harder. I have never faced a 90 mph fastball, so I have no idea why hitters find it so much harder to make contact with a 90 mph fastball than with a 92 mph fastball. But they do.

Explaining the non-linearity might be easier. Pitchers on the low end of the scale (with average fastballs in the mid 80's) only stick around in the majors by doing lots of other things well. Those that don't somehow manage reasonable strikeout rates do not keep their jobs. Pitchers on the high end of the scale (those who flash 98 mph fastballs and who sit above 94 mph) are so rare that hitters won't be used to facing that kind of heat. So strikeout rates for those pitchers should be exceptionally high.

Remember Joba Chamberlain in 2007? His average fastball speed out of the pen was 97 mph (!) according to FanGraphs. In 2008, his average fastball speed fell to 95 mph, and yet he still maintained a strikeout rate of 10.6 K/9. However in 2009, his average fastball rate dropped to 92.5 mph. Still well above average, but no longer elite. His strikeout rate fell to 7.6 K/9, which is also well above average, but no longer spectacular. Say what you want about Joba's other pitches. It's his declining fastball speed that drove down his strikeout rate.

Lefties and the National League

Low strikeout rate has got you down? Want to increase your strikeout rate by up to 2.0 K/9 in two easy steps?

First, learn how to throw left handed. Second, sign with the Nationals and make sure that you get into their starting rotation. Or join any other National League team, for that matter. An lefty starter in the National League with an above-average fastball will average almost 2.0 more K/9 than a right-handed American League starter with the same stuff, according to my model:

model:	hard throwers:	soft throwers:
starters: THROWS=L	+1.296	+0.355
relievers: THROWS=L	+0.939	+0.650
starters: LG	-0.424	-0.213
relievers: LG	-0	-0.117

Since LG = [AL = +1; NL = -1], the -0.424 value means that switching from the AL to the NL will gain a hard throwing starter almost 0.8 k/9. Since the league change effect is almost entirely absent for relievers, I think that the increase in strikeout rate is due to NL starters facing the opposite pitcher. It seems that a starter in the NL can pad his strikeout rate simply by having a good enough fastball to make it hard for the opposing pitcher to make contact. This should be worth noting for AL teams that sign hard-throwing NL starters.

If you look at Randy Johnson, Kevin Brown and Curt Schilling, all of these guys had their SO9 rates dip by about 2.0, when they made late-career moves to the AL East from the NL West. Two years later, Randy Johnson went back to the NL West, and his strikeout rate jumped right back up, despite a small drop off in fastball velocity. The other two pitchers retired.

I don't know offhand what the difference is between strikeouts in the AL and in the NL. But I'm sure that the difference is not 4.0 strikeouts per game. Whatever the difference is, it looks like it's being disproportionately made up for by the hard-throwing starting pitchers in the NL. Soft tossers like Barry Zito should not expect their strikeout rates to rise with a move to the Senior Circuit. However, Zito's left-handedness serves him well in both leagues.

(I am not implying that the three star pitchers' drop in SO9 rates proves that when hard-throwing pitchers move the AL, their strikeout rates drop by 2.0 strikeouts per nine innings. My model suggests that the drop should be more like 0.8 strikeouts per nine innings. However even a drop like that would probably not be accounted for by simple translations of average strikeout rates between the NL and AL. That's all I was saying. This sounds like a worthwhile study. And a fairly simple one, also.)

Repertoire Depth

The models seem to suggest that having a deep repertoire is negatively correlated to strikeout rates. This is not entirely so.

Although all of the model weights associated with rep_offerings and rep_depth are negative or zero, these weights are assigned in a context where we can give positive weights to pitchers for individual pitches.

I also trained models using only the most important features (here is the model for starters):

• FB velocity
• handedness
• league (AL vs NL)
• rep_depth and rep_offerings

Such models showed that (for both starters and relievers) rep_depth is positively correlated with strikeout rates, while rep_offerings is negatively correlated with strikeout rates. Also, the absolute value of the weight for rep_depth is higher in all cases.

In other words, having multiple "core" pitches is predictive of a high strikeout rate, but throwing lots of pitches, without throwing them very often, is not.

Furthermore, a model with the five features above performs almost as well as a model with the full set of features. I will explore this deeper in a future post.

As for the current models, which include features about individual pitches, why are the rep_depth and rep_offerings weights always negative? I can't say for sure. But we can't simply ignore the fact that many pitchers who have had very high strikeout rates (Gagne, Lidge, Papelbon, Clemens, Randy Johnson) were all one-pitch or two-pitch pitchers. If the model sees many examples of pitchers with really high strikeout rates that use just two good pitches, that fact will be reflected in the feature weights.

Other Pitch Stats

As I mentioned above, using the most important five stats can get us a strikeout rate model that is within a few percentage point of models trained with the full set of features. That is not to say the the other features are meaningless. The have some additional predictive value, but most of the information these features offer is redundant. However they are interesting for descriptive purposes.

By allowing the model to consider information about a pitcher's curveball and change up, we should get better estimates about how significant his fastball speed really is. However it is difficult to tell anything conclusively about these "lesser" pitches' predictive power.

The model seems to suggest that hard-throwing relievers should use the change up as a core pitch. If they are going to throw sliders, they better be hard sliders. That sounds reasonable. However I would not take these numbers too seriously. The relevant weight are here. If you see something interesting that I missed, please let me know!

Odds & Ends

This post has already run long, so I'll just mention a few more observations, without comment:

1. If we build a model for left-handed starters, we get a 0.75 correlation to the observed strikeout data.
2. If we build a model for left-handed relievers, the correlation is 0.45.
3. Giving the model features for "quality of opposition" does not help predict strikeout rates. Although I'm not sure how sensitive these features are to a pitcher's individual opposition, rather than his team's opposition as a whole.
4. The model suggests that height doesn't matter, except that it hurts lefty relievers. Higher body weight is good for relievers, but bad for lefty starters. Higher age is bad for lefty starters, but good for lefty relievers. I doubt if any of these values are significant.

Fastballs for dinner?

This post is over due.

A few months ago, I invented a new statistic to measure the depth of a pitcher's repertoire. Using such a statistic, we can say which pitches a particular pitcher relies on. Then, we can show which pitches are used by what percentage of pitchers, within the major leagues. That is what I am going to show below.

You may ask: is this different from graphing how many pitchers throw a curve at least 10% of the time? Yes, it is. Let me explain repertoire depth with a couple of examples. Or you can just skip ahead to the graphs.

-----

Take Rudy Seanez. In 2007, he threw 76 innings for the Dodgers as a right handed reliever, posting a 3.79 ERA with 1 save. According to FanGraphs pitch data, his pitch distributions were as follows:

pitch:	% thrown:
FB	53.0
SL	31.6
SF	13.0
CH	2.3

This distribution translates "2.17 depth from 3 offerings."

Offerings is a sort of upper bound on the pitcher's repertoire, while depth is a lower bound. Intuitively, the number of offerings would correspond to how many pitches a hitter should look for, while depth is a measure of how many core pitches a pitcher has. You can see the details in my original post (I'm looking at the harmonic mean of the pitch percentages). But basically, any pitch ranking lower than the number of offerings is something that the pitcher throws too rarely for us (or the hitter) to care about. While depth is a way to rank pitchers in order of their repertoire depth.

Still confused? Let's take a look at another example. Mariano Rivera is the best one-pitch pitcher in baseball. Here is how his repertoire broke down in 2008:

pitch:	% thrown:
CT	82.0
FB	18.0

That's it. He throws a lot of cutters, and also some fastballs. I have him at 1.18 depth from 2 offerings. The truth is a little bit more complicated, since I am not considering pitches thrown to RHBs and LHBs separately. Dave Allen shows that Mariano actually throws both cutters and four-seam fastballs to RHBs, but only cutters to LHBs. Then again, I don't know where to get pitch data splits for years before 2008. And I would major small sample issues.

Contrast Mariano's repertoire to that of Roy Halladay's from 2009:

pitch:	% thrown:
CT	41.5
FB	31.7
CB	22.2
CH	4.6

According to my system, his repertoire has 2.81 depth from 3 offerings. Somehow that sounds efficient, right? You don't need know a harmonic mean from a harmonica to see that the talented Mr. Halladay throws three pitches, with pretty much equal likeliness. So he's got 3 pitches, each of which are a "core pitch" for him.

----

Let's get back to those graphs I promised. For each pitch that FanGraphs catalogs for us, I count how many pitchers include that pitch in their offerings. That's simple enough. I look at his top X pitches, where X is that pitcher's number of offerings.

I do the same thing for a pitcher's repertoire depth. Roy Halladay has a depth of 2.81. Round that to 3. So we take his top 3 pitches. His three core pitches (cutter, fastball and curve) all count. Rudy Seanez (of 2007) has a depth that rounds down to 2, so we take his fastball and slider as core pitches. His splitter is #3, so it counts for offerings, but not for depth. The breakdown for Mariano is simple.

-----

If we perform the computations above for all pitcher season from 2002 to 2009, we can get a count of how many pitchers use each pitch, and how many pitcher use each pitch as a "core pitch."

We can also compare how these pitch distributions differ between starters to relievers. We all know that relievers' repertoire depths are smaller than those for starters. But how are they different? Which pitches do relievers rarely throw?

Here are the graphs. Each pitch is shown as a percentage of pitchers who include it in their offerings, and a percentage of those who include it in their depth.

And here are just the lefties (starters & relievers):

For those who love small samples, here are the lefty starters (still 318 pitcher seasons):

Alternatively, I have the same data in chart form. First, the pitches by offerings:

pitch:	% of pitchers:	% of starters:	% of lefties:	% of lefty starters:
FB	99.6	99.3	99.9	100.0
SL	72.5	70.8	72.7	63.2
CT	8.1	14.2	7.7	18.2
CB	49.5	66.7	50.7	67.6
CH	59.8	77.6	68.0	89.0
SF	8.1	10.2	1.2	2.5
KN	0.5	1.1	0	0
pitches	2.98	3.40	3.00	3.41

And similarly for depth:

pitch:	% of pitchers:	% of starters:	% of lefties:	% of lefty starters:
FB	99.3	99.0	99.9	100.0
SL	43.8	40.2	43.1	26.4
CT	4.3	7.5	3.9	11.3
CB	22.4	31.3	23.6	31.1
CH	22.2	28.1	32.0	47.8
SF	3.1	4.2	0.3	0.6
KN	0.5	1.1	0	0
pitches	1.94	2.10	2.01	2.19

Well that's a lot of numbers. You can draw your own conclusions, but a few things stick out to me:

• Starter do indeed throw more different pitches than relievers. On average, they have about 0.5 more offerings, and 0.5 more depth. Starters are more likely to use each of the different pitches than relievers. The exception is sliders. Relievers are more likely to rely on their slider than starters
• All pitchers rely on their fastball.
• Lefties throw the same pitches as non-lefties. Except that lefties are more likely to throw change-ups, and lefties almost never throw splitters. Then again, the change-ups could be a classification issue, since lefties tend to throw less hard than righties.

This article is long enough already, so I'll stop here. If you would like to see something else, please let me know.

Why strikeout rates from pitch data are interesting? (Part 2)

In Part 1, I showed that information about pitch data, bio data and league (AL vs NL) is helpful for predicting strikeout rates, but not as helpful as simply looking at the swinging strike rates. So then why should we care about predicting strikeout rates from pitch data and bio data?

I don't need to explain why predicting strikeout rates is important. But I will do so anyway. Bill James has written often (behind the pay wall) that strikeout rates are a good predictor of how likely a pitcher is to be effective in the future. There has been a trend toward higher strikeout rates for many years, and effective pitchers who don't strike people out are a dying breed. Bill James suggests that this trend will only continue. Thus it is important for teams to develop high-strikeout arms for the future. This is not to say that other factors of a pitcher's performance are not important. But strikeout projections are very important for a prospect's value to a club.

This is why I am focusing on non-performance related factors in predicting strikeout rates for major leaguers. I have never spoken to an MLB scout and I don't know much about minor league stats. But I think that the factors that I am looking at might be easy for a scout to project.

I am looking at biographical factors like:

• handedness
• height
• weight
• age

These will not change for a prospect, or at least they will change very predictably.

My pitch data includes factors like:

• how fast is his average fastball?
• how often does he throw it?
• how often does he throw a breaking ball?
• what's the speed differential between his fastball and change-up?
• how deep is his repertoire (using a statistic that I invented)?

Also, I use:

• IP (innings pitched), but only to allow the model to differentiate between starters and relievers.
• league (NL vs AL), but mostly to adjust for that fact that NL starters face the opposite pitcher, and thus have slightly higher strikeout rates that have nothing to do with ability.
• YEAR (as an number), to factor out yearly trends. However, this is almost always ignored by the model, in any case.

None of these features consider the player's results-based stats from the major league level. Also, I think that a scout could predict all of these features, at least within a range.

I would like to see my model translate a pitcher's scout projection into a projection of his future strikeout rate. And I think it can.

Say you've got Joe Dirt. His fastball sits at 92 mph, and touches 95. He's got a plus change-up, and a below average slider that he can become MLB average if he works on it. He projects as a starter. Oh yeah, and he's a lefty. Sounds like one heck of a prospect. But can we project all of that information to a MLB strikeout rate? With my model, it is possible.

Furthermore, my model should even be able to place error bars on the output. In previous posts, I have discussed how variance in predicting strikeout rates is related to IP. In a more recent post, you can see how my prediction accuracy changes by fastball velocity.

With more data and more time, I should be able to not only predict a pitcher's strikeout rate reasonably accurately, but I will also be able to say how confident I am in that prediction, based on his fastball speed, his handedness, and whether or not he throws a slider.

I would hope that such a system would be useful to scouts, and to the teams that employ them. If you are reading this and you own or run a major league team, feel free to email me. Despite the name & references to Mother Russia, I am a baseball-loving American citizen. Originally from Russia.

The system does have limitations, not least of which is the fact that my entire sample space is guys who've made the majors. I have no data on 95 mph lefties who never got to AA. I know there is much more to pitching than being a hard throwing lefty.

----

The swinging strike study (showing that swinging strike rate is very highly correlated to strikeout rate) uncovers some valuable truth, and this is not a criticism of Jeff Sullivan's work. However I'm doubtful that swinging strike rates observed at lower level will matter much for predicting those same rates at the major league level.

My own baseball career topped out at around 12 years old. I was a good Little League pitcher. I didn't throw too hard, but I got plenty of swing-and-misses from guys chasing my slow "fastballs" outside, and my "sliders" in the dirt. There is no way that my crap would have worked in high school.

Why strikeout rates from pitch data are interesting? (Part 1)

Before I go into depth about the results my findings, I think that this is a question worth answering. Why am I trying to predict strikeout rates from pitch data an biographical information?

As Dave Allen pointed out to me (thanks Dave!), one can predict strikeout rates very well with swinging strike rates. So if the point is to find a way to predict strikeout rates independent of direct measures of performance (ie wins, ERA, VORP, etc), then my gig is up. However, this isn't the point.

The article from Jeff Sullivan (linked above) shows that there is a linear relationship between strikeout rate and swinging strike rate. His R^2 = 0.71, therefore implying that the correlation coefficient between these factors is 0.84. That is much better than anything I can get from pitch data. Which I don't think is surprising. I compare these values in more detail below.

Also it's notable that Jeff uses a 100 IP cutoff for his pitcher seasons. Therefore, he completely ignores relievers. I do not ignore any pitcher seasons, as I wrote about in several previous posts. However I include my results, training only on 100+ IP pitcher seasons, as a point of comparison.

predicting:	correlation:	model type:	features:
SO%	0.84	linear	swinging strike rate
SO9	0.59	non-linear	pitch type, bio, league
SO9	0.48	linear	pitch type, bio, league
SO9	0.37	non-linear	without fastball velocity
SO9	0.52	non-linear	fastball velocity only

As I wrote before, SO% and SO9 are extremely correlated (you can estimate one from the other with a linear transformation with very high accuracy). My resulting models for SO% and SO9 are almost the same (after a linear transformation), with the same correlations. I will switch to SO% soon. I promise!

A few things are clear from the table above:

• Swinging strike rate is a much better predictor of SO% than anything available from pitch data (how hard does a pitcher throw, which pitches, and how often), bio data (height, weight, age, handedness) and the league in which he pitches.
• I gain significant predictive power from using a non-linear model. If you want to see why, take a look at the graph of fastball velocity vs strikeout rate from my previous post.
• Most of my predictive power comes from the non-linear use of the fastball velocity.

Although moving from 0.52 correlation to 0.59 correlation is significant, I must acknowledge that my model is not much more than a way to predict strikeout rates from fastball speeds, adjusted for handedness, league, age, and a few other factors. Then again, I need to re-iterate that my full model takes into account all pitcher seasons with less than 100 IP, while these models are restricted only to full time starting pitchers.

In case you are wondering what it means for my 'other features' to predict strikeout rate at a 0.37 correlation, here is an amusing parallel. If I want to predict the year in which a certain pitcher season occurred, using the same data, I get a similar correlation:

predicting:	correlation:	model type:	features:
YEAR	0.32	non-linear	pitch type, bio, league

My data consists of pitcher seasons 2002-2009. There is equal distribution of pitcher seasons among these years. The year is treated as an number, so the model gets credit for getting close to the right year, etc.

So one can predict the year that a certain pitcher season took place using a few trends in pitch data and biographical composition of the pitchers. Again, note that all pitcher seasons are for 100+ IP. The most significant features (roughly in order of importance):

• Cutter %: Pitchers throw more cutters than ever before. However, this may also reflect a bias in the classification of pitches as cutters over time by BIS. This only affects pitchers who throw cutters, whoever, who are still in the minority.
• Fastball velocity: Average fastball velocity among starters has steadily increased for the last decade.
• Repertoire depth: This is a statistic that I invented to measure how many pitches a pitcher has in his arsenal. Over the last decade, starters have developed more balanced repertoires. In other words, they throw their second & third pitches at a percentage more in balance with the percentage of fastballs thrown.
• THROWS = L: There are more lefty starters now than there used to be.
• not SO9: I allowed the model to use SO9 as a feature, but it did not find it useful to predict the year, at least not after the above factors are considered.

All of these factors add up to a 0.32 correlation with the year of the pitcher season. I thought that was an interesting list, so decided to share.

For those of you who care about details, here is how the model above (predicting pitcher year) looks like:

Rule: 1

IF

CT_fg_per > 0.05

CT_fg_per <= 9.15

THEN

YEAR_BIT =

0.0082 * THROWS=L

+ 0.0816 * HEIGHT_OVER_6ft

- 0.0004 * WEIGHT

+ 0.0359 * AGE

- 0.1365 * LG

- 0.0011 * FB_fg_per

- 0.0017 * SL_fg_per

+ 0.0657 * CT_fg_per

- 0.017 * CB_fg_per

- 0.019 * CH_fg_per

- 0.0028 * SF_fg_per

+ 0.074 * FB_fg_vel

- 0.0307 * SL_fg_vel

+ 0.0135 * CT_fg_vel

- 0.0024 * CB_fg_vel

+ 0.0012 * CH_fg_vel

+ 0.3186 * rep_depth

- 0.0047 * rep_offerings

+ 1999.0137 [314 instances]

Rule: 2

YEAR_BIT =

0.3793 * THROWS=L

+ 0.0581 * HEIGHT_OVER_6ft

- 0.0401 * FB_fg_per

- 0.0415 * SL_fg_per

+ 0.0217 * CT_fg_per

- 0.0919 * CB_fg_per

- 0.0638 * CH_fg_per

- 0.1114 * SF_fg_per

+ 0.2177 * FB_fg_vel

- 0.0356 * CB_fg_vel

+ 0.034 * CH_fg_vel

+ 1.5547 * rep_depth

- 0.2658 * rep_offerings

+ 1987.7999 [816 instances]

Strikeout rate predictions by IP buckets (visuals)

I retrained my SO9 prediction model after scaling pitcher season weights in reverse proportion to the SO9 variance implied by IP. See the previous post for an explanation of how I did this.

The model did not change much. That is good. After many iterations of training, my model has stabilized. It is now consistently splitting the pitcher seasons into three similar sized buckets. But more on that in the next post. For now, I quickly want to explain the graph above.

----

After breaking down my 5,017 pitcher seasons into 5 buckets by IP, I graph the scatter plot for actual vs predicted IP in the 5 series. Also, I graph the trend lines for each data subset.

The graph is a bit messy, but one thing is clear about the five distributions:

• The series for [+∞, 116], [116, 65] and [65, 39] IP have similar distributions
  
• The series for [14, 0] IP shows that my model has very little ability to predict SO9 rates for that series
• The [39, 14] IP series is somewhere in the middle, but closer to the first three series. So I can predict these strikeout rates somewhat well, but still much worse than those with 39+ IP.

This observation reflects the variance estimates from the previous post. Again, my new model has almost exactly the same weights as before. However since the pitcher seasons in the [14, 0] IP range have between 10 and 100 less training weight than the 200 IP pitcher seasons, the new model claims a higher correlation with the data set. I think this makes some sense.

Before I scaled pitcher seasons by inverse SO9 variance estimates, the correlation between my model and the data would swing between 0.6 and 0.4, depending on how much of the lower-IP pitcher seasons data I included. Now, the correlation stays a little below 0.6 (0.5467 in the latest model), regardless of how much of the lower-confidence data I include.

Over the several models that I trained, my ability to predict SO9 for higher-IP pitcher seasons has remained the same. However, models that are trained on 40IP cutoffs, 20IP cutoffs, etc, do very poorly in predicting IP for the pitchers with very low IP.

Therefore now, I can train on all the data, predict SO9 as well as I can for pitcher seasons of all IP, and yet not have the noisy low-confidence data make my correlations look artificially bad. So I'm happy. Now I can move on to something more interesting, like looking at the model that I'm producing more closely.

-----

While you're looking at the graph, there is one more observation worth noting:

• Even for high-confidence (high IP) data, my model consistently fails to predict high-strikeout seasons.

First of all, is this really the case? Let's zoom in to the graph for all high strikeout pitcher seasons. To avoid low-confidence points, I only plot the three series for pitcher seasons with at least 39 IP. Also to get some context and to avoid multiple-endpoint statistical traps, I plot the top 1/3 (by strikeout rates) of all those pitcher seasons. The cutoff ends up at 7.42 SO9:

As the trend lines show, my model does indeed tend to under-predict high strikeout pitcher seasons. My under-predictions increase steadily as actual SO9 increases. I especially under-predict the really high SO9 seasons.

The SO9 >= 10.0 seasons represent 19.3% of the seasons show above (which in turn represent the top 1/3 of all pitcher seasons with significant IP). So the high-strikeout pitcher seasons form a small (1/15) but significant part of my data set. If strikeout rates fluctuated wildly from season to season for individual pitchers, then this would make sense. However, intuitively we know that this is probably not the case. Pitchers like Eric Gagne, Brad Lidge, and others (see previous posts for a complete list of high-strikeout pitcher seasons) show that the ability to record exceptionally high strikeout rates is a repeatable skill.

Figuring out why I fail to predict these high-strikeout seasons is beyond the scope of this post. However I suggest some ideas why this could be the case:

1. Exceptionally high strikeout totals can not be predicted only from pitch data.
2. Relationships between pitch data and strikeout rate are not linear, therefore my piecewise-linear solutions are bound to under-estimate outputs for high-end inputs.
3. The algorithm I use to generate my models performs a smoothing operation in the data. Perhaps I am losing something valuable because of the smoothing of what look like outliers.

All three explanations make sense to me. However two of them are pretty easy to look into.

----

I retrained my model with smoothing turned off. No difference. I trained a model that only looks at fastball velocity and pitcher handedness. Again, turning off smoothing made no difference. So scratch explanation #3.

Reason #2 was a bit harder to evaluated.

First of all, I need to re-iterate that the average fastball speed (FB_fg_vel as I referenced it in early posts) is the single most predictive feature that I have. There is a 0.22 correlation between observed strikeout rate (actual SO9) and fastball speed. If I train a piecewise linear model with this single feature, I can get a correlation of 0.48. So actually, my model for predicting strikeout rate from pitch data is mostly a model for predicting strikeout rate from fastball speed.

Below, I show the relationship between fastball speed and observed strikeout rates. Also I plot my predicted strikeout rates. To make the graph less cluttered, I only plotted data from pitcher seasons with 65+ IP.

Along with the scatter plot for real and predicted strikeout rate, I also included:

• a trend line for the data, fit to a quadratic function
• a 50-point moving average of the strikeout rate (ranked by fastball speed)

Again, you can see that my predictions are much lower than actual strikeout rates for pretty much all the high-strikeout pitcher seasons. There are quite a few blue points in the top right section of the graph (>92 mph, >10 SO9) that the red points don't reach at all.

However, the trend line for my predictions and the trend line for the actual values do not differ by much, even at the high end of the fastball scale. At most, my quadratic of best fit is 0.5 SO9 below the quadratic for the actual values. The 50-point average seems to jump up for the actual values somewhere around 95 mph. But the jump isn't big enough to be sure if it's significant.

This all leads me back to reason #1. I do not have the features in my model, in order to predict really high strikeout rates. Also, I'm pretty sure I won't get there by looking at the average fastball speed on a different scale. The answer could be hidden in something I already have (but I don't present properly to the model), like the speed differential between a pitcher's fastball and change-up. Or it could be something that I haven't imported yet.

As always, the more I think I learn about baseball, the more questions I seem to raise to myself. But that is the nature of the beast.

Handling small sample sizes (K9 variance by IP)

First of all, thanks to Dave Allen for pointing out that I should be trying to predict SOr (strikeout rate as a percentage of AB) rather than SO9 (strikeout rate per 9 innings). However in practice, there is no difference. I can't give you linear a formula to translate SOr to SO9 offhand, but I can say that my machine learning system predicts them with the same accuracy (within 1% which is not remotely significant), using the same features, weighed in equal proportions. That said, Dave is right and I should be using SOr.

For the mean time, I will stick with SO9, knowing that I can translate to SOr as needed. I understand strikeout rates per 9 innings much better than I understand strikeout rates per batter. I imagine other people do as well. Seven K's per 9 innings is average. Anything around nine per nine innings is very good. Anything above that exceptional. In any case, thanks Dave. You are 100% right.

----

In my last post, I wrote about an idea I had for handling small sample sizes. Or rather, I wrote about training models intelligently with lots of data of unequal significance. I now have a much better example of this idea at work.

I have a data set of all pitcher seasons from 2002-2009, along with pitch data from FanGraphs, and the pitchers' observed strikeout rates. Also I have innings pitched (IP) for these pitcher seasons.

I would like to build an (optimal) model to predict expected strikeout rate (SO9) from the FanGraphs pitch data. However my data set includes data for guys who pitched 200 IP, and for those who guys who pitched 20 IP. Intuitively, these data points are not equally significant. However the 20 IP data points are not insignificant either! This is important since almost 20% of the pitcher season from the past decade are for guys throwing less than 14 IP in a season. I would expect strikeout rates to be very noisy at such low usage figures. I suspect that most analysts just throw data like this out. However, where do i draw the line? 20 IP? 40 IP? Maybe at a cutoff where the model looks most predictive? But that isn't very good science.

It would be nice to scale each data point by an estimte of how confident we are in the accuracy of that observation. In other words, if we can get a relative measure of variance (in our dependent variable) based on the input data (independent variables), we can scale by the inverse of that variance in training. I have a more detailed explanation of this idea in my previous post.

As before, I isolate the independent variable as IP. My dependent variable is observed strikeout rate (SO9). I estimate the variance in SO9 by looking at the error of a model I built to predict SO9 rate where all pitcher seasons are trained with equal weight. The logic here may seem circular, but there is nothing inherently illogical about it. If I simply looked at variance in observed SO9 within bands of pitcher seasons (by IP), I would be way over-estimating variance in the high IP cases. CC Sabathia, Randy Johnson, Jon Lester and Kirk Reuter will have large difference in observed SO9 rate, even over 200IP. Comparing observed rates to reasonable, unbiased estimates is a better way of trying to understand which part of the difference can be attributed to small sample sizes, and which part of the difference is due to other factors, like talent, pitching style, etc.

----

OK, so on to the graph! My data points are buckets of pitcher seasons, grouped by IP, and represented by the average IP of the bucket. There are 5,000 pitcher seasons in my sample. Each data point represents a bucket of 1,000 pitcher seasons, except for the outermost points, which represent only 500 pitcher seasons. How's that for sample size!

As you can see, the curve is modelled quite well by a f(x) = ax^(-b) function, where x = IP and f(x) estimates the variance in SO9 rate.

The next step is to re-train my model, having weighed the data points by the inverse proportion of this estimated variance. This should result in a higher correlation between real and predicted SO9 rate, using cross fold validation or any other method where I leave out part of the data for testing.

Also, I should expect to perform better on weighting-neutral tests. For example, I should more often be closer to the observed strikeout rate than my current model, which treats all the pitcher season the same.

Phrased differently, I am now training a model that places more significance on nailing down strikeout rates that I think that I should be able to predict more accurately. So being off by 2 K/9 on Roy Halladay will be less acceptable than being off by 2 K/9 on Jonathan Albaladejo. I think that makes a lot of sense!

Hopefully this approach will work. We all need better tools for tackling sampling size issues. Regression to the mean can screw up good analysis, if not properly accounted for.