Perspective on adjusted plus - minus poll

[Crow]

Some people try to characterize the APBRmetrics board or "crowd" as adjusted plus - minus devotees. I don't think that is really the case but what say you to this poll question?

[DSMok1]

I certainly feel, when properly handled, Adjusted Plus Minus is very important. It is, in my opinion, the only true independent, non-biased measure of a player's impact. Admittedly noisy, though--handle with care.

[Dr Positivity]

I like it but don't use it as a primary way of judging players. I generally find its value is showing which players correlate with success for the team, but not which players are the best in a vacuum. Two recent examples being Odom and Deng's amazing scores IMO signifying the "glue effect", where their systems got Mario mushrooms when they get on the floor, but that doesn't mean they're the best player on the floor, just perhaps the most indispensable piece.

To use a random example of coffees and different combinations, my ranking out of 10 on how much I like each kind

Black - 5: I'll drink it but I'm not happy about it
Cream, no sugar - 7: Only mildly disappointed if I have to drink this kind
Sugar, no cream - 1: Always tasted disgusting to me
Cream and sugar - 10: Delicious

Doing math on this equation, cream is a +2 when there's no sugar, and +9 when there is sugar, for a total score of +11. Sugar is a -4 when there's no sugar, and a +3 when there is cream, for a total score of -1. Coffee itself is only +5. These results are of course crazy. The cream isn't worth twice as much as coffee and cream isn't negative value or else I would never put it in.

The way to fix the scores of course is to adjust for the fact that sugar with no cream never has to happen. Even if I'm in a situation with no cream but want coffee, I will choose black instead of black with sugar. This means cream is +2 when I don't have sugar and +5 when have sugar (since the alternative is black, not black with sugar), for a total score for cream of +7. Sugar's score is neutral when I don't have cream (since I will not use sugar in that case, just picking black) and +5 when I have cream, for a total score for sugar of +5. This is exactly right, since cream SHOULD have higher value than sugar, based on the fact that I get more value out of cream without sugar, than I do sugar without cream.

Even going further though, in reality there's almost no difference between cream and sugar to my personal enjoyment, based on the fact that 99% of the time, I have both cream and sugar available to me. And when that's the case, both are worth +2.5, splitting the difference between black (5) and one with both cream and sugar (10). But let's say there was a scientific/psychological study that said when a coffee with cream is sugar with delicious to a person, surprisingly 80% of this comes from the sugar's influence and not the cream's. So instead of a split of +2.5/+2.5, it's +4 for sugar/+1 for cream, when both are added to the coffee. This would mean that to me, because of how often I have both sugar and cream, sugar is much more important to my enjoyment.

Bringing it to APM, one of my concerns is there's no way to evaluate this. Take the example of Stockton and Malone. Recent data has 2001 Stockton scoring much better. It's within reason if we had APM scores for the 90s, Stockton would be shockingly out in front of Malone. But what if Stockton is cream (superior if sugar isn't there, than sugar without cream) and Malone is sugar (when combined with cream, takes a much greater weight of the combo's success). Isn't the latter much more important, due to how much of the Jazz success relies on both their cream and sugar being there? It would seem nearly impossible to me for APM to successfully separate Stockton and Malone in value because their value is based on what they do on the court together and what the ingredients baked in the cake or in the coffee add up to - and there is no way to split up who's responsible for those team results when both were on the court. Likewise with Deng and Rose or Odom and Bryant, there's no way to know whether Deng and Odom were just that good, or whether they were the hypothetical cream in the situation, a compliment to sugar that creates a massive swing in value, even though the majority of that value is in reality because of sugar's deliciousness.

[DSMok1]

Excellent points, Dr. Positivity. Wouldn't this in effect also apply to box-score metric, also? Interaction effects yielding the actual numbers questionable in regards to player valuation?

If we want to estimate a player's value, there aren't too many ways to go about it.

[Bobbofitos]

Really loved the coffee metaphor... Had never thought of APM in that light.

Voted "these are not my views at all" because I am unclear if we're dealing with APM or RAPM. I put a lot of stock in the latter, almost none in the former...

[mystic]

Dr Positivity, your example has a big flaw, because it is based on a subjective scale. While you can use that to illustrate a point, that issue makes it tough to convert the problem to basketball. The score is objective, when a team scored +5, they scored +5, not +7 or +3, because you liked or didn't like the way they scored. A seperation between two players with high amount of minutes can only happen during the time when they played without eachother. And that's where the issues with APM are coming in, when the sample is small. Due to those multicollinearity issues in the ill-posed problem we see big differences in the results based on overfitting. That's why RAPM is preferable over APM.

The other issue in regarding to Odom and Deng is: There is no vacuum, there is only a 5on5 game happening. People are trying to distinguish a player from the rules of the games, if they are trying to argue that we have to look at the best in a vacuum. The setting is there, the rules are there, basketball is played in such an environment. That players will look different in a different environment (or even vacuum) doesn't change the rules of basketball. But when evaluating players it seems as if too many people are rather trying to change the environment before accepting that their own preconception is actually the issue here, not the method per se.

To put the coffee example in here: We are not comparing coffee black with coffee+cream or coffee+sugar or coffee+sugar+cream, we are actually comparing different coffee+sugar+cream. Some people like a different sweetner than sugar, some people like milk instead of cream (just a different water+fat emulsion), essentially the same things just with small variations. Or maybe a different coffee from a different company. Overall all those 3 things are always included, coffee+sweetner+emulsion, like in a NBA basketball game, where always 5 players play against another 5 players. That there are different varieties (like 1on1 games, HORSE, 3on3, etc.) doesn't really matter for the problem at hand, because we only want to know how good a player is within the 5on5 environment of the NBA.

Overall a team needs both, players who can put numbers (preferable in a efficient way) into a boxscore while also making the team work in order to achieve the main goal, outscoring the other team. Thus, I use both, boxscore-based and +/- based analysis to judge a player. Well, and given the retrodiction test stuff, that seems like the best way to predict out of sample. I selected that option, assuming that APM is meant here as big sample or RAPM based analysis.

[xkonk]

Not to put words in Dr. Positivity's mouth, but I think he was referring specifically to the interactions and combinations inherent to basketball, not using coffee to point out that APM is somehow subjective. Cream in one context (with sugar) is 'worth' one value but in another context (without sugar) may be worth something different. Basketball players are the same. Using APM or RAPM, it is unclear where a player's value comes from; is it because he's objectively good at something on his own, or is it because he brings something to the players he plays with that boosts the team/line-up? I'm not sure that Jerry's comparison RAPM even quite answers the question. Looking at Dirk last year (http://stats-for-the-nba.appspot.com/pm/5.html), is he simply 10 points better than Cardinal? Or is he maybe 6 points better and his teammates play 4 points better as well compared to when those guys play with Cardinal? Could he be 12 points better but his teammates actually play 2 points worse, and Cardinal is a 'glue guy' who's simply outclassed by Dirk's abilities? Without explicitly modeling the interaction effects it's hard to say. Using the boxscore to add more information helps, but as Daniel pointed out it is still subject to some of the same questions.

[Dr Positivity]

Well bringing it back to Deng vs Rose again. We know Deng correlates well with the Bulls' caliber of play. However what's actually happening could be two very different causes:

a) Deng has a superstar impact on the game, period. He makes the Bulls better because of how good he is and the difference in rep between Rose and Deng is just a fault of overvaluing offense and ppg vs defense

b) When Deng is on the floor, he makes other players like Rose, Noah, Gibson, etc. more valuable. Thus the CAUSE of Deng's +/- stats is not Deng, but the talent of Rose, Noah, Thibs system, and the rest of the Bulls, or w/e.

The distinction between the two is very important. In an example where the Bulls were deciding whether to trade Deng for an offensive star like say, Chris Bosh (who scores worse in RAPM), if a) was correct, the trade would be poor for them because they are getting a worse player. If it's b), it might be a great trade for them because Deng in this case may be replaceable, perhaps if they sign Landry Fields they retain 80-90%+ of Deng's value, the real original cause of Deng's high RAPM scores (Rose, Noah, Gibson, Thibs, etc.) are still on the team

+/- is based on outcomes (lineup data), but the issue is because there is 5 players on the floor at all times, we can never really know who's responsible for each piece of lineup information/outcome. In the coffee example only a hypothetical scientific study, could reveal that 80% of the reason sugar/cream has more value than black coffee (an outcome) is because of sugar, while isolating one from the other favored cream. Whereas because a 5 man unit has 1 lineup outcome, it is impossible to actually weigh its factors in this same fashion

The best RAPM can do, is gather enough outcomes and lineup data to make a very good guess on who causes these outcomes the most, but clearly this has issues due to players being on one team virtually all the time, and playing much of the same minutes with other star/starter players. I think RAPM has a TON of value in regards to correlation, but not causation

[DSMok1]

APM measures which of those two cases are actually occurring, does it not?

APM looks at the Bulls' performance, with vs. without Deng, with the other 4 held steady. So if it is Deng that is the cause, they should be worse without him and with someone else. If it is a matter of those 4 having a good synergy and just needing any SG/SF to plug into the 5th spot, that will show up too.

(With a large enough sample size, of course).

This page looks at those cases directly: http://stats-for-the-nba.appspot.com/pm/122.html

Deng was 10 points better than Korver with the same 4 man units, and 5 points better than Brewer. He was equal to Butler, though with a smaller sample size.

Context certainly matters, I'll grant you that, though. Perhaps (likely) Korver works better with one 4 man unit and Deng better with another.

[Dr Positivity]

APM measures which of those two cases are actually occurring, does it not?

APM looks at the Bulls' performance, with vs. without Deng, with the other 4 held steady. So if it is Deng that is the cause, they should be worse without him and with someone else. If it is a matter of those 4 having a good synergy and just needing any SG/SF to plug into the 5th spot, that will show up too.

Yes and no. My example of trading for Bosh and replacing Deng with Fields wasn't a great one because Brewer may be a decent sign whether Fields could replace Deng. A better one would be, say Toronto wanted to trade for Luol Deng. This would be putting him in a vastly different system. Whether Deng was replaced well by Brewer and Korver wouldn't help Toronto much in this decision to decide whether Deng would be a superstar for them, or simply the straw that stirs the drink of Rose, Gibson, Noah, Thibs system, etc.

Putting it another way, presume we're talking about these two lineups

Lineup A
Rose
Rip
Deng
Gibson
Noah

Lineup B
Rose
Rip
Korver
Gibson
Noah

The one with Deng plays significantly better, supporting the RAPM numbers of a top 15 player. In regards to "cause", I see 3 questions that need to be answers

Cause Question 1: When Lineup A is on the floor, who is most responsible for Lineup A's performance
Cause Question 2: When Lineup B is on the floor, who is most responsible for Lineup B's performance
Cause Question 3: Who is responsible for the difference between Lineup A and Lineup B's performance

RAPM specializes in answering Cause Question 3. Answering Cause Question 1 and Cause Question 2 is impossible using +/- data because all 5 players have one output value. RAPM uses some very, very good methods to try as hard as possible to turn the results of different Cause Questions 3 into answers for the first 2, but there's still a gap there. Deng can correlate by far the best lineup wise with success but not be having the largest impact on the game when he's in. Deng could answer Question 3 with flying colors, but in reality, whenever he plays with Rose and the bigs, they're impacting the game more than him. If true, Deng passing Question 3 with flying colors is very misleading. The important part for players is how much they're impacting the game period, when they're playing, IMO. A player being the straw that stirs the drink is extremely important for the Bulls to know who their important players are, but it's not as useful to determine who is actually the best player, from the perspective of judging players in a vacuum or what they would do if another team acquired them

[xkonk]

Moving away from Dirk/Cardinal or Deng/Korver, let's take an example of two putatively equal players. Last year Garnett and Collison rated out very similarly on RAPM (if you wanted to use 2 year +/-, we could say Collison and Bosh or Aldridge). Let's say we were going to play a game now, where you get first pick and can take one of those two. I then get the other one, but I get to pick the other four guys for your team. Would you be indifferent between the two? My impression is that many people would rather have Garnett (or Bosh or Aldridge). Even assuming that the equal rating is accurate, it feels like Garnett's is due to things that he brings to the court and could do with many different groups of teammates while Collison's is due to the particular players structured around him. That is, I think I could pick teammates that would easily sabotage your Collison team while it would be harder to sabotage your Garnett team. Given who they actually tend to play with, however, they are equally valuable.

[DSMok1]

The one with Deng plays significantly better, supporting the RAPM numbers of a top 15 player. In regards to "cause", I see 3 questions that need to be answers

Cause Question 1: When Lineup A is on the floor, who is most responsible for Lineup A's performance
Cause Question 2: When Lineup B is on the floor, who is most responsible for Lineup B's performance
Cause Question 3: Who is responsible for the difference between Lineup A and Lineup B's performance

RAPM specializes in answering Cause Question 3. Answering Cause Question 1 and Cause Question 2 is impossible using +/- data because all 5 players have one output value. RAPM uses some very, very good methods to try as hard as possible to turn the results of different Cause Questions 3 into answers for the first 2, but there's still a gap there. Deng can correlate by far the best lineup wise with success but not be having the largest impact on the game when he's in. Deng could answer Question 3 with flying colors, but in reality, whenever he plays with Rose and the bigs, they're impacting the game more than him. If true, Deng passing Question 3 with flying colors is very misleading. The important part for players is how much they're impacting the game period, when they're playing, IMO. A player being the straw that stirs the drink is extremely important for the Bulls to know who their important players are, but it's not as useful to determine who is actually the best player, from the perspective of judging players in a vacuum or what they would do if another team acquired them

What do we mean by "who is most responsible for Lineup A's performance"? The lineup is, basically, a single entity with a single response variable (point differential).

We want to know which player is most valuable and attempt to quantify performance for, in essence, out-of-sample prediction.

Thus, we are trying to disambiguate the value of a player in certain contexts.

We have as data the performance of the team in certain contexts, as quantified by the box score and (especially) the scoreboard.

Could we assume, that, as the number of contexts and sample sizes go toward infinity, the answers for the above questions will converge? I think the answer is yes.

[DSMok1]

Moving away from Dirk/Cardinal or Deng/Korver, let's take an example of two putatively equal players. Last year Garnett and Collison rated out very similarly on RAPM (if you wanted to use 2 year +/-, we could say Collison and Bosh or Aldridge). Let's say we were going to play a game now, where you get first pick and can take one of those two. I then get the other one, but I get to pick the other four guys for your team. Would you be indifferent between the two? My impression is that many people would rather have Garnett (or Bosh or Aldridge). Even assuming that the equal rating is accurate, it feels like Garnett's is due to things that he brings to the court and could do with many different groups of teammates while Collison's is due to the particular players structured around him. That is, I think I could pick teammates that would easily sabotage your Collison team while it would be harder to sabotage your Garnett team. Given who they actually tend to play with, however, they are equally valuable.

A very insightful post, I think. Fit and synergy is key, but really, really hard to measure or quantify. We can only measure the specific contexts that have occured, which are but a small subset of the possibilities.

The whole concept of "position" gets at this issue, but only in a very clumsy way in my opinion.

[Mike G]

.. Even assuming that the equal rating is accurate, it feels like Garnett's is due to things that he brings to the court and could do with many different groups of teammates while Collison's is due to the particular players structured around him. ... Given who they actually tend to play with, however, they are equally valuable.

Is it "equally valuable" to maintain this value over 15-20 minutes or for 30-35 mpg?
Whether Collison's minutes are limited more by his beneficial-teammate specificity, or by his high foul rate, how can he be as valuable as a guy who can play twice as long?

[mystic]

Not to put words in Dr. Positivity's mouth, but I think he was referring specifically to the interactions and combinations inherent to basketball, not using coffee to point out that APM is somehow subjective.

I have a hard time putting this statement by you into relation to the points I made. At no point did I say that Dr. Positivity's post would imply some sort of subjectivity in APM, but that HIS subjective scale for coffee is actually screwing the thing up here. It is very likely that he knows beverages which taste worse than coffee+sugar and some which taste better than coffee+sugar+cream, thus making his grades 1 respective 10 very suspicious.

And if you read my 2nd paragraph again, you might notice that I actually don't deny the existing of interactions or combinations or "glue", just that I said that this is PART of the game, something which might just be intrinsic to the player, not something we have to seperate here.

As I see it, people who are trying to seperate interaction effects from the players are at fault here, because the game is played 5on5 and the effects have to be seen as PART of the equation, not something we have to exclude or evaluate differently. A player who brings that "glue" to the team, is actually valuable, and at that we want to know how valuable he as a player within the framework of a basketball game is. If we remove the "glue", we are actually ignoring PART of the game. That is the mistake a lot of people are actually making when evaluating players.
We are not interested in knowing how good a player would be in a different setting (e.g. 1on1 game or 3on3, etc pp.), we are interested in the value he can bring to the team within a 5on5 game. To understand that is imho one really important step towards understanding the decision making by the NBA teams.

An NBA team can't just use some sort of statistical metric based on production or efficiency in order to come up with a decision, because those metrics aren't telling us anything about team fit or interaction effects. A player with a lower metric can actually be more helpful in a specific situation, because he actually makes it possible to get others into a better position on the court, helping them to achieve better results and finally achieving a better overall team result. Your Nowitzki/Cardinal example is really illustrating that. If Nowitzki in terms of production/efficiency is just about +6 better than Cardinal, but his team plays actually +10 better, the 4 additional points are created by the fact that Nowitzki makes it easier for his teammates. That is part of the equation here, that is part of the intrinsic value of Nowitzki. We have to take that into account, not trying to seperate that from the player. If Cardinal has that "glue" value, so be it. It is really nothing we have to worry about, because it might just be something based on Cardinal's skillset and personality, which is a trait specific to Cardinal in that case.

Using APM or RAPM, it is unclear where a player's value comes from; is it because he's objectively good at something on his own, or is it because he brings something to the players he plays with that boosts the team/line-up?

Those "all in one" metrics suffer the same problem, because they are not there in order to seperate among player's skills, but rather trying to appoint a value to a player for the overall game results. If we just have the boxscore based metric in front of us, we also have NO information where the value actually comes from. In order to answer that, we have to look deeper into the boxscore values of the player while knowing what value those numbers have within the framework of the specific boxscore based metric. The question is: What value has such metric, if we can't answer something which seems to be an important question? The same thing is true for +/- based metrics or the combination of +/- and boxscore based metrics. The part of skill, fit, glue is not answered, but that is in fact not a seperate issue within the 5on5 environment of basketball, it is PART of the game. And it is an important part a boxscore metric can say nothing about.

Using the boxscore to add more information helps, but as Daniel pointed out it is still subject to some of the same questions.

For me boxscore based metrics are important, because they tell us something which is important to the game of basketball. Someone has to put those numbers into the stats sheet, because a lineup with 5 glue guys will most likely not achieve much. On the other hand 5 high productive guys will likely also not achieve much, if they can't find a common basis on which they can work together in order to achieve overall team success. So, in order to find a better overall metric, we might actually need to blend boxscore based metrics with +/- based metrics. And I guess it was shown during those retrodiction tests, that the blended metric is outperforming the seperate metrics. I think that can be used as a proof that either way is missing something. My conclusion, and that was something I found already a couple of years ago, that relying only on one specific value (like a boxscore based metric (e.g. PER) or +/- based metric (e.g. APM)) will actually give a least accurate description of the value a specific player brings to the team. And something which seems really important: my blended SPM+RAPM values from 2004 to 2012 are showing a higher year-to-year correlation than either SPM or RAPM seperately.

As Mike pointed out already, your Collison vs. Garnett example failed to take another important information into account: minutes played. Those minutes might be allocated based on several different things like depth at the position, fit, endurance, foul trouble or match up. Players playing more minutes might just be more versatile in order to find fit, favorable matchups or just foul less while having better endurance or there is just simply a void behind them on the team due to injuries or an overall lack of talent on the specific position. So, being able to play minutes seems to say something about the value of the respective player to the specific, maybe even more than any other boxscore entry can say. So yeah, I take Garnett who is able to keep his level for a longer time, while also having a distinct advantage in terms of boxscore production.

Well, and as the poll shows, the people here seems to rely much less on a single number than people somewhere else. Taking all evidence into account that seems to be the far smarter way than relying entirely on a suspect boxscore based metric.