Sanity checking my RAPM methodology for 2019-2022 seasons

[jc114]

Hi,
Thanks for taking the time to read this! I thought I'd take the time to compute RAPM myself for fun and I wanted to get some feedback and sanity checking. My methodology is as below:

1. Download NBA play by play data, using defensive rebound, turnover, end of clock, made shots to split possessions and convert to stints. Tried to be careful here but there's definitely some cases I didn't handle.
2. Filter out 0 possession stints and non-NBA teams.
3. Adjust for home-away advantage, roughly add 0.02 PPP to away offensive possessions. (Future: FT%, 3PT%, rubber band adjustments)
4. Replace players that played less than 1% of the teams possessions with a single id and filter out the stints which only contain these players.
5. Double and flip each stint to calculate offensive and defensive RAPM separately. Remove the replacement level players with the single id and use the value -0.02. Recompute the PPP of each stint by subtracting the league average PPP and clipping to [-3, 3].
6. Sparse L2 regression with sample weight = num possessions of stint * season weight. (Season weight: 2021-2022=3/6, 2020-2021=2/6, 2019-2020=1/6 to weight recent seasons more). Tried alpha=500,1000,1500
7. Print out and sanity check (alpha=1000): https://docs.google.com/spreadsheets/u/ ... MP/pubhtml

The biggest issue seems to be the scale of Defensive RAPM vs Offensive RAPM is very sensitive to the league average. I used average of stints PPP ~= 1.13 but the ratio of defensive/offensive RAPM of the top players is roughly 0.04/0.02. The average ORAPM is -0.025 while the average DRAPM is 0.025. If I switch to a league average of 1.1, the offensive numbers start to look better. Should I be enforcing the mean of ORAPM to be 0 and DRAPM to be 0 (i.e. add 0.025 to ORAPM and subtract from DRAPM)?
With respect to the rankings, it seems ok-ish but Nikola Jokic and Lebron James and James Harden are probably too low and Gobert, Quickly and Muscala are too high.

[Mike G]

Hey, thanks for sharing. Will take some time with it later.
Meanwhile, why not have your table per 100 possessions? Maybe convert .0736676325 to 7.36 ?
(I doubt you are claiming that much accuracy.)

[jc114]

Here is the converted one! https://docs.google.com/spreadsheets/d/ ... q4/pubhtml

[J.E.]

The results look pretty sane, so good job

1. Download NBA play by play data, using defensive rebound, turnover, end of clock, made shots to split possessions and convert to stints. Tried to be careful here but there's definitely some cases I didn't handle.

Don't forget And1s

3. Adjust for home-away advantage, roughly add 0.02 PPP to away offensive possessions. (Future: FT%, 3PT%, rubber band adjustments)

Adjusting for 3pt% never actually did anything - it's just a marketing term for clowns that don't understand data science

4. Replace players that played less than 1% of the teams possessions with a single id and filter out the stints which only contain these players.

Completely unnecessary, especially since it's 2023 and 32G of RAM cost <$100

5. Remove the replacement level players with the single id and use the value -0.02. Recompute the PPP of each stint by subtracting the league average PPP and clipping to [-3, 3].

Clipping? What? All of it unnecessary, see above

The biggest issue seems to be the scale of Defensive RAPM vs Offensive RAPM is very sensitive to the league average. I used average of stints PPP ~= 1.13 but the ratio of defensive/offensive RAPM of the top players is roughly 0.04/0.02. The average ORAPM is -0.025 while the average DRAPM is 0.025. If I switch to a league average of 1.1, the offensive numbers start to look better. Should I be enforcing the mean of ORAPM to be 0 and DRAPM to be 0 (i.e. add 0.025 to ORAPM and subtract from DRAPM)?

I generally center both offense and defense seperately, so that the sum of all minute-weighed {offensive, defensive} ratings equals 0. Not sure if it helps in this specific situation
You can try different alphas for offense/defense, if your implementation of Ridge allows for it. Offensive alpha usually lower, allowing for more movement away from 0

[jc114]

The results look pretty sane, so good job

Thank you!

1. Download NBA play by play data, using defensive rebound, turnover, end of clock, made shots to split possessions and convert to stints. Tried to be careful here but there's definitely some cases I didn't handle.

Don't forget And1s

Ah I forgot, good catch!

3. Adjust for home-away advantage, roughly add 0.02 PPP to away offensive possessions. (Future: FT%, 3PT%, rubber band adjustments)

Adjusting for 3pt% never actually did anything - it's just a marketing term for clowns that don't understand data science

I see. My impression is this should help with predicting for low possession players, but wouldn't be surprised if the actual impact is minimal.

4. Replace players that played less than 1% of the teams possessions with a single id and filter out the stints which only contain these players.

Completely unnecessary, especially since it's 2023 and 32G of RAM cost <$100

I didn't do this for memory costs - I did this because estimates for low usage players would be inaccurate/unstable anyway so we may be better off replacing them with static values.

5. Remove the replacement level players with the single id and use the value -0.02. Recompute the PPP of each stint by subtracting the league average PPP and clipping to [-3, 3].

Clipping? What? All of it unnecessary, see above

I implemented clipping to account for data processing mistakes.

The biggest issue seems to be the scale of Defensive RAPM vs Offensive RAPM is very sensitive to the league average. I used average of stints PPP ~= 1.13 but the ratio of defensive/offensive RAPM of the top players is roughly 0.04/0.02. The average ORAPM is -0.025 while the average DRAPM is 0.025. If I switch to a league average of 1.1, the offensive numbers start to look better. Should I be enforcing the mean of ORAPM to be 0 and DRAPM to be 0 (i.e. add 0.025 to ORAPM and subtract from DRAPM)?

I generally center both offense and defense seperately, so that the sum of all minute-weighed {offensive, defensive} ratings equals 0. Not sure if it helps in this specific situation
You can try different alphas for offense/defense, if your implementation of Ridge allows for it. Offensive alpha usually lower, allowing for more movement away from 0

Got it. Let me try this - thanks for the suggestions!

[jc114]

I ended up making those rubber-band and FT%, 3PT% adjustments over the weekend. Results are here: https://docs.google.com/spreadsheets/d/ ... Zz/pubhtml
The results are a little different. Are they better? Really hard to say. Jokic is 1st here but Lebron has dropped more. Some guys like Quick are lower but then Brook Lopez is higher. idk.

[J.E.]

I didn't do this for memory costs - I did this because estimates for low usage players would be inaccurate/unstable anyway so we may be better off replacing them with static values.

Ridge penalization makes these not so unstable already, at least in an absolute sense

I implemented clipping to account for data processing mistakes.

Should probably sanity check outside the regression code, then take a hard look at your parser if something's wrong.
3 is a weird number to clip at. Probably hurting Curry and everyone that's good at 3pt-And1s, or good FT offensive rebounders

The results are a little different. Are they better? Really hard to say

Wrong. You see, the whole beauty of using this specific technique is to be able to truly say whether one metric outperforms another.
You can easily compute these ratings on a daily basis, then use the ratings of day X to predict lineup performance at day X+1
Whatever metric gets you lower sum of squared error of (actual_pts - expected_pts), is the better one

[jc114]

I didn't do this for memory costs - I did this because estimates for low usage players would be inaccurate/unstable anyway so we may be better off replacing them with static values.

Ridge penalization makes these not so unstable already, at least in an absolute sense

Yes, L2 does do that but that doesn't mean preprocessing is bad.

I implemented clipping to account for data processing mistakes.

Should probably sanity check outside the regression code, then take a hard look at your parser if something's wrong.
3 is a weird number to clip at. Probably hurting Curry and everyone that's good at 3pt-And1s, or good FT offensive rebounders

I could certainly clip to a different number but I don't agree that is weird. What number would you suggest?

The results are a little different. Are they better? Really hard to say

Wrong. You see, the whole beauty of using this specific technique is to be able to truly say whether one metric outperforms another.
You can easily compute these ratings on a daily basis, then use the ratings of day X to predict lineup performance at day X+1
Whatever metric gets you lower sum of squared error of (actual_pts - expected_pts), is the better one

This is a good point and its true that this can be measured in standard time-series way. Its also worth pointing out that there are likely to be tradeoffs to weigh once you do things like bucketize and then analyze MAE or MSE or any other metric. What happens if one method does better on one metric but worse on another? Its maybe a bit too bold to claim that a metric solves all problems.