Predictive Stats

[deepak]

EDIT (DSMok1): This discussion has been split off of the Translating WS/48 to PER thread.

Win shares are probably the best commonly available box-score metric for prediction, certainly very superior to PER.

What kind of prediction? If this is true, I guess the main reason is because team defensive rating is a not-so-insignificant component of it.

[v-zero]

Prediction of game/season outcomes for rosters.

[DSMok1]

Win shares are probably the best commonly available box-score metric for prediction, certainly very superior to PER.

Yes, I agree. My ASPM is likely at least as good as Win Shares, also.

[Neil Paine]

Win shares are probably the best commonly available box-score metric for prediction, certainly very superior to PER.

Yes, I agree. My ASPM is likely at least as good as Win Shares, also.

From what I've seen & researched on my own for ESPN, ASPM is slightly better than WS48 at predicting future seasons, and both are far above Wins Produced and PER. The more years out you go (trying to predict year N+2, N+3, etc from individual stats in year N), the closer PER gets to WS48 -- to the point that, in terms of predicting 3 years into the future, WS48 and PER are essentially equally accurate (PER never catches ASPM, though). RAPM is consistently better than any boxscore metric. WP48 is better than PER at predicting N+1, but falls well behind all other metrics (PER included) at predicting N+2, and is absolutely blown away by the other metrics at N+3.

[Mike G]

From what I've seen & researched on my own for ESPN, ASPM is slightly better than WS48 at predicting future seasons, and both are far above Wins Produced and PER. The more years out you go (trying to predict year N+2, N+3, etc from individual stats in year N), the closer PER gets to WS48 -- to the point that, in terms of predicting 3 years into the future, WS48 and PER are essentially equally accurate (PER never catches ASPM, though). RAPM is consistently better than any boxscore metric.

Neil, are you describing how these metrics predict individual players' future metrics? -- or how the player metrics predict team strength?

[v-zero]

Yes, I agree. My ASPM is likely at least as good as Win Shares, also.

No offence intended, I put 'commonly available' in as your ASPM isn't as pervasive as WS just yet.

[Neil Paine]

From what I've seen & researched on my own for ESPN, ASPM is slightly better than WS48 at predicting future seasons, and both are far above Wins Produced and PER. The more years out you go (trying to predict year N+2, N+3, etc from individual stats in year N), the closer PER gets to WS48 -- to the point that, in terms of predicting 3 years into the future, WS48 and PER are essentially equally accurate (PER never catches ASPM, though). RAPM is consistently better than any boxscore metric.

Neil, are you describing how these metrics predict individual players' future metrics? -- or how the player metrics predict team strength?

This is how they predict future team strength.

Basically, if I know the distribution of minutes for every 2012 team, but I only know the players' PER/ASPM/WS48/WP48s from 2011, which metric should I choose if I want to most closely predict team wins in 2012? (Assigning league-average rates to rookies and low-minute players from 2011.) Ok, then how about if I only know the players' PER/ASPM/WS48/WP48s from 2010? How about 2009?

Do that for every season since 2001, and you find that RAPM is the best predictor, followed by ASPM, then WS48/PER (WS48 does much better if I need to predict only 1 season into the future, but the two metrics equalize the further into the future I try to predict), followed by Wins Produced at the bottom of the totem pole.

[Kevin Pelton]

The more years out you go (trying to predict year N+2, N+3, etc from individual stats in year N), the closer PER gets to WS48 -- to the point that, in terms of predicting 3 years into the future, WS48 and PER are essentially equally accurate.

Doesn't this tend to suggest that a lot of the success Win Shares and Wins Produced have in predicting year N+1 is due to team defensive ratings?

I've been thinking about this a bit lately, and what I'd love to see is prediction broken down by percentage of returning minutes. If my supposition is correct, Win Shares/Wins Produced will dominate for teams with little turnover, but may not be better for teams that have made over their rosters.

Really, the ideal for comparing rating systems in terms of prediction is an expansion team, but unfortunately we can count that sample on one hand in the last two decades.

[xkonk]

Neil - Which 'flavor' of RAPM did you use for that work? The most recent xRAPM version, the one that carried the previous season all the way forward as a prior, the one that used 0 as a prior...?

[talkingpractice]

In our testing of all of the open source metrics before building our own IPV, prior informed RAPM (using last year RAPM as prior) > ASPM >>>>> everything else (by a mile). Nothing else was really close to those two. And ASPM is as good as RAPM if you're only concerned with the offensive side of the equation (and maybe even better).

IPV is so predictive because it has the benefits of both (prior informed RAPM for defense, and a RAPM model with spm prior for offense).

If considering only open source metrics, then something like RAPMd + 0.75*ASPMo + 0.25*RAPMo would be the nuts.

I don't know why anyone would use offensive WS as opposed to ASPMo.

[Neil Paine]

Neil - Which 'flavor' of RAPM did you use for that work? The most recent xRAPM version, the one that carried the previous season all the way forward as a prior, the one that used 0 as a prior...?

It's the version J.E. currently has on his site, which I believe is informed by previous seasons' ratings as a prior.

Doesn't this tend to suggest that a lot of the success Win Shares and Wins Produced have in predicting year N+1 is due to team defensive ratings?

I've been thinking about this a bit lately, and what I'd love to see is prediction broken down by percentage of returning minutes. If my supposition is correct, Win Shares/Wins Produced will dominate for teams with little turnover, but may not be better for teams that have made over their rosters.

Really, the ideal for comparing rating systems in terms of prediction is an expansion team, but unfortunately we can count that sample on one hand in the last two decades.

I agree -- it does seem that ratings with heavy team-based defensive adjustments lose their effectiveness quickly, probably because whatever defensive magic the team was working is not being captured in the players' boxscore stats (and therefore isn't portable in different contexts).

I've long advocated the idea of weighting retrodictive accuracy by the "continuity" of the team being retrodicted. It does appear to be much easier to use previous stats to project performance for teams that hardly changed personnel. When I was testing linear weights metrics for Prospectus, I broke it down by the continuity index (% of returning MP) of the teams being predicted:

http://www.basketballprospectus.com/art ... cleid=1985

Alternate Win Score (Dan Rosenbaum's Wins Produced modification) seemed to retain its effectiveness the most under any circumstance, although none of the metrics being tested included a team adjustment.

[Kevin Pelton]

I probably should have remembered that, seeing as I edited it and all!

[DSMok1]

Neil - Which 'flavor' of RAPM did you use for that work? The most recent xRAPM version, the one that carried the previous season all the way forward as a prior, the one that used 0 as a prior...?

It's the version J.E. currently has on his site, which I believe is informed by previous seasons' ratings as a prior.

xRAPM (which was current until J.E., as it appears, was hired by an NBA team), used boxscore data to inform the prior, along with prior year data. See this thread for a discussion, for those who weren't following all of the developments: http://www.apbr.org/metrics/viewtopic.p ... 5&start=45 .

[Guy]

Doesn't this tend to suggest that a lot of the success Win Shares and Wins Produced have in predicting year N+1 is due to team defensive ratings?

I've been thinking about this a bit lately, and what I'd love to see is prediction broken down by percentage of returning minutes. If my supposition is correct, Win Shares/Wins Produced will dominate for teams with little turnover, but may not be better for teams that have made over their rosters.

Really, the ideal for comparing rating systems in terms of prediction is an expansion team, but unfortunately we can count that sample on one hand in the last two decades.

I agree -- it does seem that ratings with heavy team-based defensive adjustments lose their effectiveness quickly, probably because whatever defensive magic the team was working is not being captured in the players' boxscore stats (and therefore isn't portable in different contexts).

In the case of Wins Produced, I don't think it's only the team adjustment that explains the relative success in predicting teams with low turnover. The high valuation of rebounds will also be a big plus, because that captures a lot of a team's aggregate defensive performance. As long as the players stay the same, WP will capture defensive value via rebound totals, even though it badly misallocates value among players. But once the roster changes, that will be lost quickly since the high-rebound players are not necessarily those creating the missed opponent shots. (Ironically, the "new WP" with a lower coefficient for DReb, while perhaps a better stat, is likely to do worse at predicting N+1 team wins.)

Also, WP's failure to account for usage won't be a handicap unless/until you have roster turnover. The overvaluing of low-usage players is simply offset by undervaluing of high-usage players. Until the roster changes, this weakness won't be exposed.

[Neil Paine]

In the case of Wins Produced, I don't think it's only the team adjustment that explains the relative success in predicting teams with low turnover. The high valuation of rebounds will also be a big plus, because that captures a lot of a team's aggregate defensive performance. As long as the players stay the same, WP will capture defensive value via rebound totals, even though it badly misallocates value among players. But once the roster changes, that will be lost quickly since the high-rebound players are not necessarily those creating the missed opponent shots. (Ironically, the "new WP" with a lower coefficient for DReb, while perhaps a better stat, is likely to do worse at predicting N+1 team wins.)

In the case of non-team adjusted stats (like my basic linear weights test), a heavy DReb weight will absolutely lead to more "predictive" accuracy (whether split-season, N+1, etc). But that's just because DReb's are defensive stops, so they serve as a proxy for the team's actual defensive quality. Among stats with a team adjustment such that the sum of player ratings = team quality, I'm not sure if the heavier DReb weight will lead to more predictive accuracy in the short term. I suppose it's possible, for high-continuity teams in particular, but my assumption was that the defensive team adjustment levels the playing field between metrics in that regard.