The Flaws of Synergy for Defensive Rankings

[EvanZ]

I like the thought grey. I'm trying to find another situation where two players usually sub in for each other, and one of them has produced really poor results, and it creates an odd-looking APM for the other guy.

Probably Andris Biedrins and Ekpe Udoh, which we now may have a good test since the latter was traded to Milwaukee.

[mystic]

I think we're still talking separate things. I dunno, maybe not.

Evan was saying Bonner's RAPM isn't necessarily due to sampling error, which I'm fine with. "indeed not much to do with sampling errors" is a different thing entirely.

Uh, maybe my wording isn't the greatest and from a philosophic point of view those statements are not the same, but in the end I meant the same.

You're talking about an issue with measurement accuracy, and that's somewhat separate from variance, which is a randomness in sampling issue.

I was talking about both, the measurement accuracy and variance. When I said that the players aren't consistent, I mean that there is variance in their performance level.

It sounds like you're assuming the sample size of 66 games is plenty big enough, meaning there is basically zero randomness and no variance. IMO most of the over and underrating in a non-prior APM comes from 66 games being way too small a sample, which produces large coefficient errors.

For normal APM I completely agree. Take the Lakers in 20010/11 and look what normal APM makes with Gasol and Bryant. That tells us that even 82 games are not enough. Well, even with the 66 games added from this season normal APM is basically overrating Gasol and underrating Bryant. That is just an issue with overfitting. And that is basically eliminated in RAPM. Even the non-prior informed version has both players much closer together in 2011 and 2012. For RAPM the sample size can be a lot smaller in order to have not big issues with multicollinearities like that. There are really much, much different in that aspect.
So, if you are only talking about APM, you are correct, but somehow I had the impression we are talking about RAPM here.

[xkonk]

Not to drag the thread increasingly away from Synergy, but how do we know that RAPM has eliminated overfitting? The cross-validation certainly reduces it compared to APM, but how would we ever know it's gone?

[EvanZ]

Not to drag the thread increasingly away from Synergy, but how do we know that RAPM has eliminated overfitting? The cross-validation certainly reduces it compared to APM, but how would we ever know it's gone?

The only thing I can think of is to compare to LASSO and see which one produces lower C.V. error. Sometimes that works, but it can produce some funny results.

[xkonk]

I guess my question was more broad than that even - how would we ever know if there was no overfitting for any prediction method (including LASSO)? Wouldn't it have to generate perfect predictions? Otherwise aren't the errors a pretty impenetrable combination of model mis-specification, overfitting, noise, and so on?

[EvanZ]

I guess my question was more broad than that even - how would we ever know if there was no overfitting for any prediction method (including LASSO)? Wouldn't it have to generate perfect predictions? Otherwise aren't the errors a pretty impenetrable combination of model mis-specification, overfitting, noise, and so on?

Well, I think if the out-of-sample and in-sample error are similar, then overfitting is probably not a major issue. If OOS error is significantly larger, than overfitting may be a factor.

[mtamada]

I guess my question was more broad than that even - how would we ever know if there was no overfitting for any prediction method (including LASSO)? Wouldn't it have to generate perfect predictions? Otherwise aren't the errors a pretty impenetrable combination of model mis-specification, overfitting, noise, and so on?

Well, I think if the out-of-sample and in-sample error are similar, then overfitting is probably not a major issue. If OOS error is significantly larger, than overfitting may be a factor.

It's an interesting idea, I suspect though that what it would really tell us is that the sample size is so small that the in-sample stats should not be trusted to be valid out-of-sample. Not so much due to over-fitting, but due to, as xkonk suggests, over-fitting plus any of a wide variety of other causes in particular noise due to small sample size.

It's possible that in the situation described, i.e. estimates which show significantly larger OOS errors, the estimates are nonethless the best ones obtainable, over-fitting or no. So although the larger OOS errors would warn us to be cautious when forecasting, we couldn't conclude that over-fitting had occured nor that the model was bad. Larger OOS are a fact of life, it's nice when they're only slightly larger rather than a lot larger but often that's out of our control.

[Crow]

In addition to Evan's A4PM detail, there is Jerry's RAPM pair data. Another tool to shine light into the black box". For 2011 Bonner has a non-negative rating with every Spur on offense and defense except he couldn't save the pair with Gary Neal on defense. Knowing RAPM for individuals and RAPM pairs, one can also calculate the RAPM for a player without another player. There is plenty of detail that can be worked up and consulted if one is curious about it. For example Bonner-Ginobli was estimate the 11th best player pair on offense in the league in 2011.

If there was enough interest, it would be possible to calculate same 4 other player RAPM at factor level and the same for pair with and without cuts and take all of them easily to direct / indirect splits and perhaps down into further detail using nonboxscore data. If you did all this one could have up to 64 or more RAPM based angles of observation on a player beyond the boxscore. Looking at all of them carefully, with the boxscore and the video, there would be a lot of input available to synthesize a very detailed understanding of a player and where it appears he did his best work and where his weaknesses showed up. If one or two RAPM ratings is too "blackbox", use 16, 64 or more. Even if you don't trust the estimates much or at all for various reasons, they can suggest things to research further by other means and even that is useful.

[schtevie]

I just want to take a half step back and address the text and sub-text regarding the apparent overvaluation of Matt Bonner (a similar discussion of Vince Carter could follow). It seems to me that a conversation on a topic such as this would be very well served by having an explicit baseline of comparison, in particular one generated in the most straightforward way from box score statistics.

Let's leave aside the defensive side of things and focus on offense. Why? Because this is an area where box score stats can speak most clearly and because what drives the "OMG, Matt Bonner get's what rating?!?" discussion is Jeremias' 2012 number on the offensive end. So, these are the Offensive RAPM numbers for MB's career:

2005: -0.5
2006: -0.4
2007: -2.4
2008: -2.6
2009: +0.4
2010: +1.5
2011: +2.1
2012: +4.4

The issue, it seems to me, is the degree to which these RAPM numbers are out of line with a conventional reading of box score numbers. As I peruse the advanced stats on Basketball Reference, it isn't obvious to me that there is going to be that much of a residual to explain in 2012. And my impression is that the changes over time broadly correspond to the movement in the RAPM rating.

This year (regular season) MB had the lowest TOV% by far for all forwards having played 20 or more mpg, and he had the 9th highest TS%. These two facts are a very big deal in box score land.

(Yes, his ORB% was no great shakes, but this statistic is of lesser import offensively than the other two - as you cannot get an offensive rebound if you've already turned the ball over or scored. And as for assists, my quick read is that he assisted about as much as he was assisted, so this shouldn't net to much, however you want to treat this "speculative" statistic.)

I don't know if Bob Chaikin is interested in the task, but it would interesting to see how his simulation estimates Matt Bonner's yearly offensive contributions, relative to the "average" forward. My guess is that a straight up simulation would show the contribution of Matt Bonner's 2012 offensive box score stats to be not far from +4.4 points per 100 possessions.

[Mike G]

Yeah, but could 5 Matt Bonners beat 5 Chris Pauls?

[EvanZ]

If we chose MVP to be "a team consisting of 5 of this guy would be the best", wouldn't LeBron be the obvious winner? I don't think it would even be close with anyone else.

[schtevie]

Plastic Man would kick all their butts.

[Crow]

Bonner's RAPM changed fairly dramatically in 2008-9 from the previous season on both offense and defense. What explanations can be provided and what team changes can be considered for their impacts?

On offense he shot and scored at his then career best efficiencies in 2008-9. He cut his turnover rate in half from the previous season. On the other hand his usage went down by 25%.

Barry left, Roger Mason arrived. By RAPM they were both neutral players in their respective years so that probably didn't have an impact in comparison to Bonner but maybe.

Bowen had his minutes cut by 40% and slipped from below average to well below average. Maybe that affects Bonner's relative rating.

Ginobili's minutes in 2008-9 were only half of the previous season while Bonner's went up to 225% of the previous season. He went from playing with Duncan and Parker 35-45% of his minutes to 60-70%. He apparently thrived in that new context and without Ginobili there to grab a lot of the credit for the performance of those units. His time shared with Parker and Duncan is back down to 30-40% but maybe he and the team learned how to better utilize him during Ginobili's absence. And Horry was gone in 2008-9.

Between 2007-8 and 2008-9 both Parker and Duncan's RAPM each fell by over 2 pts. Bonner and others got more of the credit. What changed in the team strategy / style of play? It is not immediately obvious to from the overall efficiencies and factor ratings and shot distributions and eFG%, which seem about the same. Average league offense efficiency was it is high for recent times and thus defensive efficiency was at its recent low. Maybe that was conducive to Bonner's game.

With more research, more can be found though. He went from 2 lineups used over 22 minutes to 13 and from a top minute lineup of 70 minutes to 540. He played in more familiar lineups. That appears to have helped.

Coincidence with league averages or whatever but a clear majority of the top 20 overall RAPM rankings in recent years have been posted by guys who were moderate to high frequency 3 pt shooters.

[Crow]

Of the 60 or so guys who are +2 or better on prior-informed RAPM there are maybe half a dozen guys whose ratings would draw serious question from some, in my estimate. 10%. That ain't much. That probably should be expected. That probably shouldn't be the main basis for dismissing RAPM.

[schtevie]

Well, here's my effort in trying to determine what, if anything, needs to be explained regarding Matt Bonner's Offensive RAPM over the course of his career. The exercise is to determine the direct contribution he has made (relative to the median NBA Forward) in not turning the ball over and scoring, attribute these effects entirely to him, then to see how the sum of these two factors compare to Jeremias' offensive numbers. (Again, I don't include offensive rebounding, primarily because it isn't going to influence things much, but a discussion could be had on that point.)

Some particulars: To determine the median forward referent, I queried the B-R database assuming an 18.9 mpg or above (MB's career average). As for the counterfactual value of turnovers, I used the team average offensive rating.

To not bog things down with too many numbers, let me just summarize the general effect of MB's better ball possession and scoring years. Basically, in his low TOV% years, given his usage rate, this is worth about a point. Then, in his more efficient TS% years, this is worth about two points.

But let me display three relevant summary statistics, by year. The first is a repeat listing of the Offensive RAPM. The second is the notional direct effect of (non)turnovers and scoring. The third is the residual (Off. RAPM minus the aforementioned sum). This is an estimate of all the indirect effects that RAPM captures, relative to a "hard" box score baseline:

2005: -0.5, +3.3, -3.8
2006: -0.4, +2.3, -2.7
2007: -2.4, +0.8, -3.2
2008: -2.6, -0.6, -2.0
2009: +0.4, +3.2, -2.8
2010: +1.5, +2.3, -0.8
2011: +2.1, +2.9, -0.8
2012: +4.4, +2.9, +1.5

So, poor Matt Bonner. With the anomaly of his first two years in SA, where for reasons that might be simply told, when his "direct" contributions faltered, his primary box score contributions have been very positive (about +3). Then what you see in the "residual" is more or less what we should think normal, in terms of career progression. Specifically, rookies enter the league not knowing how to play NBA basketball, showing a significant, negative "indirect" contribution, then learn over time.

Why poor Matt Bonner? After what appears to be a typical career progression in terms of expected gains from experience, this has been the first year where the residual comes in positive, at what is roughly the expected peak of his career, and all of a sudden, it's kick a ginger time.

It may be, could well be, that the +1.5 residual is too high. Debit this by whatever guess one wishes, but the bottom line, it seems to me, is that the onus is on those who think that this year's performance was anomalous.