APBRmetrics

The discussion of the analysis of basketball through objective evidence, especially basketball statistics.
It is currently Fri Dec 26, 2014 6:33 pm

All times are UTC




Post new topic Reply to topic  [ 2 posts ] 
Author Message
PostPosted: Fri Apr 15, 2011 1:25 am 
Offline

Joined: Thu Apr 14, 2011 11:10 pm
Posts: 2683
page 1 of 3

Author Message
Dan Rosenbaum



Joined: 03 Jan 2005
Posts: 541
Location: Greensboro, North Carolina

PostPosted: Wed Dec 12, 2007 12:17 am Post subject: The Pot Calling the Kettle Black Reply with quote
At long last here is a working paper version of my paper with Dave Lewin that we presented at the NESSIS conference at the end of September. As we mention in the acknowledgments, this paper has benefited greatly from comments from many of you. And this is still a working paper, so we would love to hear reactions to the paper. Feel free to post your comments here or e-mail them to me. Please let me apologize in advance if I am not able to respond quickly to comments, but each of my three jobs right now keep me pretty busy (and that doesn't include my two year-old).

http://www.uncg.edu/eco/rosenbaum/nessis.pdf

The Pot Calling the Kettle Black: Are NBA Statistical Models More Irrational than “Irrational” Decision-Makers?

Abstract
Recent research suggests that “statistics and analysis” typically lead to better decisions than “intuition and human intellect” in diverse areas such as choosing which students to admit to college and assessing mortality risks among cancer patients. Sports economics, with its rich data and abundant decisions to analyze has provided a fertile laboratory for studies of the efficiency of decision-making. In fact, researchers of the National Basketball Association (NBA) have used statistical models of player productivity to make strong claims about the “rationality” of NBA decision-makers. Yet these statistical models have rarely been subjected to any rigorous examination of their ability to forecast the future. We examine how well several player productivity metrics, including (a) John Hollinger’s Player Efficiency Rating, (b) Wages of Wins Wins Produced, and (c) the NBA Efficiency metric, do in predicting future team wins and future player productivity (the latter as measured by plus/minus statistics). In addition to a comprehensive examination of the player productivity metrics used by NBA statistical analysts, this paper is the first academic presentation of plus/minus statistics. Our findings provide a counterweight to much of prevailing literature and suggest that models that assume simplistic NBA decision-making often outperform more sophisticated statistical models.
Back to top
View user's profile Send private message Send e-mail Visit poster's website Yahoo Messenger
mikez



Joined: 14 Mar 2005
Posts: 75


PostPosted: Wed Dec 12, 2007 11:10 am Post subject: Reply with quote
Even ignoring its discussion of the "Is Wins Produced Flawed" question, this is an important paper and a significant contribution to the community. Thanks guys.

(Also thanks Cavs, for allowing them to continue to publish despite the work they're doing for the team.)

-MZ
Back to top
View user's profile Send private message
Westy



Joined: 15 Nov 2007
Posts: 15
Location: Chicago, IL

PostPosted: Wed Dec 12, 2007 4:00 pm Post subject: Re: Pot Kettle Black Reply with quote
Hmm, I'm very interested to see the response this paper receives.

With the post from TrueHoop, you can certainly anticipate it will have some readers.

What is the plan for future publication (i.e. how long will it remain a 'working' paper)?
Back to top
View user's profile Send private message
Dan Rosenbaum



Joined: 03 Jan 2005
Posts: 541
Location: Greensboro, North Carolina

PostPosted: Wed Dec 12, 2007 4:42 pm Post subject: Reply with quote
My guess is that the reaction will be less than you think. The issues here are subtle and technical, which makes them easy to ignore. There is the potential for a really interesting and substantive discussion to take place, but based upon past experience I highly doubt that is what will take place. But we'll see. Hopefully I am wrong.

I also want to point out that Berri should get some credit for the way in which his work has motivated deeper thinking about how to evaluate players. I sincerely mean that.
Back to top
View user's profile Send private message Send e-mail Visit poster's website Yahoo Messenger
Harold Almonte



Joined: 04 Aug 2006
Posts: 616


PostPosted: Wed Dec 12, 2007 4:50 pm Post subject: Reply with quote
That's true. A lot of detractors think like his book is some kind of Davinci code of basketball metrics, some followers think it's really the apocriphal gospels, but it's just science procedures from different hiphotesis that needed to be tested. I think this paper is the test.
Back to top
View user's profile Send private message
NickS



Joined: 30 Dec 2004
Posts: 384


PostPosted: Wed Dec 12, 2007 5:29 pm Post subject: Reply with quote
Dan Rosenbaum wrote:
I also want to point out that Berri should get some credit for the way in which his work has motivated deeper thinking about how to evaluate players. I sincerely mean that.


I've said this as well, and equally sincerely. I think that DeanO and Kevin Pelton are correct that it's unfortunate if discussions of player ratings swamp other APBRMetric topics, but I think that WoW has sparked a number of interesting questions.
Back to top
View user's profile Send private message
Eli W



Joined: 01 Feb 2005
Posts: 402


PostPosted: Wed Dec 12, 2007 5:53 pm Post subject: Reply with quote
Dan Rosenbaum wrote:
My guess is that the reaction will be less than you think. The issues here are subtle and technical, which makes them easy to ignore. There is the potential for a really interesting and substantive discussion to take place, but based upon past experience I highly doubt that is what will take place. But we'll see. Hopefully I am wrong.


Unfortunately it looks like Dan was right. Here's Berri: "It is essentially the same as the powerpoints. And it has the same problems. Go back and read the thread on the Shane Battier post. I have nothing to add to what I said then."

http://dberri.wordpress.com/2007/12/12/ ... ment-50391
_________________
Eli W. (formerly John Quincy)
CountTheBasket.com
Back to top
View user's profile Send private message Send e-mail Visit poster's website
HoopStudies



Joined: 30 Dec 2004
Posts: 705
Location: Near Philadelphia, PA

PostPosted: Wed Dec 12, 2007 7:13 pm Post subject: Reply with quote
Eli W wrote:
Dan Rosenbaum wrote:
My guess is that the reaction will be less than you think. The issues here are subtle and technical, which makes them easy to ignore. There is the potential for a really interesting and substantive discussion to take place, but based upon past experience I highly doubt that is what will take place. But we'll see. Hopefully I am wrong.


Unfortunately it looks like Dan was right. Here's Berri: "It is essentially the same as the powerpoints. And it has the same problems. Go back and read the thread on the Shane Battier post. I have nothing to add to what I said then."

http://dberri.wordpress.com/2007/12/12/ ... ment-50391


My take on Dan's comment was that it should spark substantive discussion among us.

Read it carefully. I can't say that I "got it" in Boston and need to read the paper to really understand what is going on. I'd think that everyone here would have to as well.
_________________
Dean Oliver
Author, Basketball on Paper
The postings are my own & don't necess represent positions, strategies or opinions of employers.
Back to top
View user's profile Send private message Visit poster's website
Eli W



Joined: 01 Feb 2005
Posts: 402


PostPosted: Wed Dec 12, 2007 7:27 pm Post subject: Reply with quote
HoopStudies wrote:
My take on Dan's comment was that it should spark substantive discussion among us.


Could be. I took the "past experience" part to be referring to Berri. Either way, it looks like I spoke too soon. Berri has followed up with some additional comments on the paper.
_________________
Eli W. (formerly John Quincy)
CountTheBasket.com
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Harold Almonte



Joined: 04 Aug 2006
Posts: 616


PostPosted: Wed Dec 12, 2007 7:44 pm Post subject: Reply with quote
What they say about Dan including the +/- stuff in the end could be valid, it hurts a bit the paper, but it's his paper.
Back to top
View user's profile Send private message
Dan Rosenbaum



Joined: 03 Jan 2005
Posts: 541
Location: Greensboro, North Carolina

PostPosted: Wed Dec 12, 2007 10:21 pm Post subject: Reply with quote
Here apparently is the argument that Berri has against what we are doing in our paper.

Dave Berri wrote:
You cannot use residuals to evaluate a model. Rosenbaum claims that the residual from a team efficiency model is the same as the “team adjustment” used in Wins Produced. It is not. The team adjustment is not a residual. And you cannot, and I repeat you cannot, use a residual to evaluate a model. Team efficeincy regressed on points scored does not result in a model that explains very much of team efficiency. To say that with the residual the model can explain efficiency is meaningless. Any collection of independent variables plus the residual would explain a dependent variable. So Rosenbaum’s basic approach is incorrect.

This is a really bad argument. There are any number of ways to show that.

(1) Berri states that "you cannot use a residual to evaluate a model." In other words, you cannot construct a model using the residual in period T and evaluate how well that model explains the dependent variable in period T. I completely agree with that point and we make that point in the paper. But that is not what we are doing. We are constructing a model in period T with the residual in period T and using that to predict the dependent variable in period T+1, T+2, or T+3. If there is an obvious problem with doing that, Berri should be able to easily show the math behind the problem or point to a textbook or economics paper that shows what is wrong with doing this. Berri has never done so, so his argument boils down to "trust me, because I am right." And in a practical sense, there is no problem here. If it is summer 2007 and the GM is trying to predict performance in 2007-08, the residual from 2006-07 is known, so there is no problem with him using that to form predictions about 2007-08. Berri is using a straw man argument by implying that somehow we are somehow using the 2007-08 residual here, which we clearly are not doing.

(2) Berri is also using a residual to create Wins Produced. Look at equation (A.1) in the paper. Once Berri aggregates Wins Produced for a team, he gets PTS(T) from points for individual players, PTS(O) from the opposing team, and POSS(T) and POSS(O) from a combination box score stats from individual players and the opposing team. But POSS(T) minus POSS(O) does not add up to zero like it needs to in (A.1), so Berri estimates team rebounds by "add[iing] together the constant term and error term [my italics] for each team" (WoW, p. 243). The "residual" that he uses to construct team rebounds results in POSS(T) minus POSS(O) equaling zero. I have no objection to what Berri is doing, but it he can use a "residual" in creating Wins Produced, it seems odd to say there is problem with someone else using that same approach.

(3) In the appendix of the paper, we describe how each of the team adjustments can be constructed without ever using a "residual." Adapting Berri's treatment of assists allows any metric to have a team adjustment that is exactly equal to the dreaded "residual," but is constructed simply by allocating team statistics to players in the same manner Berri does. It may be a different set of statistics, but otherwise it is the same as what Berri is doing.

(4) Alternate Win Score is derived in precisely the same manner as Wins Produced (although rounded values are used like in Win Score), except that instead of assuming different production functions for own team possession production and opponent possession production, the same production function is assumed for both and alpha is assumed to be 0.7 (whereas Berri assumes it is 0 in one case and 1 in another). Berri never defends his assumption that how possessions are produced on one side of the court is completely different than how they are produced on the other side of the court. Possessions are produced in a joint production function involving all ten players on the court at any one time, and it is hard to justify why that production function depends on what side of the court the ball happpens to be on.

Alternative Win Score is the same as Wins Produced, except that coefficients on missed shots (field goals and free throws) and rebounds are altered. It is grounded in theory every bit as much as Wins Produced. There is no need to resort to a "residual" to create Alternate Win Score. Alternate Win Score performs dramatically better than Wins Produced. The difference in correlations with the given outcome (team wins or adjusted plus/minus) is statistically significant at the 4% level in on comparison and 0.3% or less in the other four comparisons. These empirical results strongly support the finding that Wins Produced is theoretically flawed.

Quote:
And one more point (which I also made before), Rosenbaum shrugs off the inability of his plus-minus model to forecast. But since that is how he is evaluating all models (even if his evaluation is very, very flawed) it is odd that his model does not come close to passing his one standard.

Wins Produced performs poorly both with the team wins outcome and the adjusted plus/minus outcome. So even ignoring the adjusted plus/minus results, Wins Produced performs very poorly. Measurement error or "noise" in a dependent variable is not a problem as long as (1) the "noise" is independent of the independent variable and (2) the "noise" isn't so great that it eliminates all of the power of any hypothesis tests from the regression. Both of the conditions are met in this case, so there is no problem with using adjusted plus/minus as a barometer.

Using the one-year-at-a-time version of adjusted plus/minus as a predictor is a separate question that has nothing to do with its validity as a barometer. I got several suggestions to remove the adjusted plus/minus as a predictor stuff from the paper, but I did not think that was a good idea. It is not good practice to hide results that are inconvenient, especially in this case where the inconvenience comes from folks misunderstanding of a widely known fact about "noisy" dependent variables. That is probably not something that typically comes up in undergraduate econometrics courses, so that might be why Berri does not appear to understand this point. [Ironically, it was Dave - an undergraduate - who had me insert a footnote on this point in the paper.]

Finally, this criticism of adjusted plus/minus would be even more true of using team wins as a predictor of player value. According to Berri's mistaken argument, this somehow would invalidate team wins as a barometer.
Back to top
View user's profile Send private message Send e-mail Visit poster's website Yahoo Messenger
Dan Rosenbaum



Joined: 03 Jan 2005
Posts: 541
Location: Greensboro, North Carolina

PostPosted: Wed Dec 12, 2007 10:41 pm Post subject: Reply with quote
HoopStudies wrote:
Eli W wrote:
Dan Rosenbaum wrote:
My guess is that the reaction will be less than you think. The issues here are subtle and technical, which makes them easy to ignore. There is the potential for a really interesting and substantive discussion to take place, but based upon past experience I highly doubt that is what will take place. But we'll see. Hopefully I am wrong.


Unfortunately it looks like Dan was right. Here's Berri: "It is essentially the same as the powerpoints. And it has the same problems. Go back and read the thread on the Shane Battier post. I have nothing to add to what I said then."

http://dberri.wordpress.com/2007/12/12/ ... ment-50391


My take on Dan's comment was that it should spark substantive discussion among us.

Read it carefully. I can't say that I "got it" in Boston and need to read the paper to really understand what is going on. I'd think that everyone here would have to as well.

No, I was referring to Berri. In this light it is useful to note the contrast between Berri's dismissal of this forum and Justin Wolfers' praise of it.

jwolfers wrote:
With apologies for being behind the times on this discussion, I just wanted send a quick note to say that this discussion thread has struck me as a particularly careful and well-informed assessment of my research with Joe Price.

There are obviously quite subtle statistical issues at play here, and also some difficult issues of interpretation, and folks here seem to have been admirably careful in actually reading the paper and thinking about the relevant statistical, economic and basketball issues.

Also, I wanted to say thanks to the folks in this forum for contributing quite significantly to my understanding of basketball statistics over the years. I have been a longtime lurker here, and have found the commentary here to be incredibly useful.

Justin Wolfers

If you look up the Justin's Google Scholar cites, his articles add up to over 1,500 cites with 10 papers with 20 or more cites. Berri's total comes to about 200 cites (over a longer career) with only 1 paper with 20 or more cites. Berri is a very solid economist, but Justin is a superstar in the profession.

As I said earlier, Berri should get credit for sparking conversation. And he has done a fabulous job translating the findings of the economics literature to the masses. But I still think his avoidance of this forum speaks volumes when folks like Wolfers seem to find some value here.
Back to top
View user's profile Send private message Send e-mail Visit poster's website Yahoo Messenger
Dan Rosenbaum



Joined: 03 Jan 2005
Posts: 541
Location: Greensboro, North Carolina

PostPosted: Thu Dec 13, 2007 1:35 am Post subject: Reply with quote
Berri makes claims like this all of the time.

Quote:
I understand your frustration. There is a proper third party. And that is a peer-reviewed refereed journal. Dan can talk about this paper at the APBR forum forever. But that is not a proper blind reviewed forum. My sense is that most people posting there do not understand econometrics well enough to evaluate his argument. Plus, it is not blind review. As JC Bradbury told Dan when he was posting anonymously here — submit your paper to an economics journal and let other economists evaluate your work properly. That is how this should be done. In fact, that is what should have happened a long time ago. I have a real problem with releasing working papers to a crowd and then declaring, before any academic has seen your work, that you have discovered something. That is not how academic research should be done (or in most places, is done).

It should be noted that Berri has never published a paper detailing the Wins Produced method in a peer-reviewed journal. Books and book chapters are not peer-reviewed in the same way that peer-reviewed journals are. And it is just not true that academics don't make working papers available to the general public. Papers from the NBER Working Paper series (where Justin and Joe Price first made their race-study available to the public) probably get as much attention from academics and especially policymakers as the entire collection of peer-reviewed journals combined. Many high profile academic papers accumulate half or more of their cites before they are ever published. A paper of mine with my advisor at Northwestern was cited more than 100 times before it was ever published. Getting published in a good peer-reviewed journal is a critical validation for an academic, but a lot of times it is somewhat of an afterthought. The real impact of a paper often comes long before a paper is published.

I think this issue of putting work out in the public before it is fully baked is a BIG part of why Berri and I have butted heads. This kind of thing is fairly common in the high profile policy areas in applied micro-econometrics where I have made my mark, but my sense is that sports economists tend to keep close hold on their papers until they are published. And I pushed the envelope even further by presenting results in a forum like this, which I think infuriated Berri. And he is not completely unjustified in feeling this was unfair, because often posts here on something this complex lack the detail and context to give Berri a chance to respond to it.

I like working out loud, even though it is dangerous, because it is easy to look like a fool when you haven't throught things through enough. But the other side of the coin is that working out loud allows for collaboration with a large number of people in real time and that can be a great way to make advances, especially when a lot of the folks who have thought the hardest about issues are non-academics.

Berri criticizes the econometrics ability of many of the folks here and some of that is legitimate. But I have also tended to find that many folks here have remarkably good econometrics intution - better than a lot of Ph.D economists. And I think I am somewhat of an authority on this in that I have taught applied micro-econometrics at the undergraduate, MA, and Ph.D levels and for the past decade I have been interacting with the superstars in the area of applied micro-econometrics. My publication record is mediocre, but I have a good reputation for knowing what I am talking about in the area of applied micro-econometrics. That is how I landed this plum position at the White House Council of Economic Advisers. (It certainly wasn't my political views.)

But I think Berri immediately got angry with me for working out loud. Then he assumed that I ghost-wrote Roland Beech's review of Wages of Wins, which I did not. But at that point, he got his guard up and it was hard to engage him publicly or privately in any substantive discussion of Wins Produced. In desperation I have tried posting anonymously a couple times, because that seemed to be the only way to get a substantive response. That was stupid on my part on many levels, but I have to laugh when I get accused of "promoting my own work" anonymously. I get WAY too much attention and hardly need any more; it didn't even cross my mind that posting anonymously would end up being interpreted that way. With my blinders on I was just trying to engage Berri.

But that is no excuse. Berri wanted to have this discussion on his terms and I should have respected that. In the long-term that would have led to a healthier relationship and more substantive discussion.
Back to top
View user's profile Send private message Send e-mail Visit poster's website Yahoo Messenger
Analyze This



Joined: 17 May 2005
Posts: 364


PostPosted: Thu Dec 13, 2007 1:39 am Post subject: Reply with quote
I like the pot calling the kettle black because it's not your typical I have found a metric and will defend it to dead article. You look critical to some metrics including your own and try to be objective.

However I find the division between "irrational" traditional decision makers and "new"decision makers/statistical analysts a bit to strong. I think that you have a third party namely the nba decision maker who uses his traditional experience/intuition on one hand and stat analysis/metrics on the other hand while he realizes the strenghts and limitations of the metrics he uses and the context he needs to see it within and also the advantages and shortcomings of the more traditional view. I think that if you combine that traditional approach with what the best of new stat analysis has to offer and you place it in the right context, you see what it tells you and what not, that you are a better decision maker than when a) you only lean on the traditional way of evaluating talent or b) you only use statistical analysis to judge players productivity.

With the article you also risk that people who are against stat analyis will use your words to attack stat analysis as a source. They can say look the analyst himself shows that complicated stat models are sometimes less accurate than an easy/ traditional way of looking at it. You see this stat anlysis is worthless. Of course that is not what your article is about but in a world were slogans mostly prevail above text with nuance it could happen. However that should never stop you to write a critical article like you did. After basketball on paper it was the best thing i read.
_________________
Where There's a WilT There's a Way

Last edited by Analyze This on Thu Dec 13, 2007 1:49 am; edited 1 time in total
Back to top
View user's profile Send private message
Dan Rosenbaum



Joined: 03 Jan 2005
Posts: 541
Location: Greensboro, North Carolina

PostPosted: Thu Dec 13, 2007 1:41 am Post subject: Reply with quote
Analyze This wrote:
I like the pot calling the kettle black because it's not your typical I have found a metric and will defend it to dead article. You look critical to some metrics including your own. However I find the division between "irrational" traditional decision makers and "new"decision makers/statistical analysts a bit to strong. I think that you have a third party namely the nba decision maker who uses his traditional experience/intuition on one hand and stat analysis/metrics on the other hand while he realizes the strenghts and limitations of the metrics he uses and the context he needs to see it within and also the advantages and shortcomings of the more traditional view. I think that if you combine that traditional approach with what the best of nba analysis has to offer and you place it in the right context, you see what it tells you and what not, that you are a better decision maker than when a) you only lean on the traditional way of evaluating talent or b) you only use statistical analysis to judge players productivity.

Exactly. You are preaching to the choir on this point.

page 2

Author Message
jamsesh



Joined: 12 Dec 2007
Posts: 6


PostPosted: Thu Dec 13, 2007 3:39 am Post subject: Reply with quote
Dan, I hope you to send this to an economics journal or whatever Dave wants you to do. Not that I think it is necessary, but if that is what it will take to get a response from Dave I say just go for it. While you may feel like its not necessary, if it forces a difference response from Dave I feel like you will be very pleased you did it.
Back to top
View user's profile Send private message AIM Address
Guy



Joined: 02 May 2007
Posts: 128


PostPosted: Thu Dec 13, 2007 6:47 am Post subject: Reply with quote
Dan Rosenbaum wrote:
Berri wanted to have this discussion on his terms and I should have respected that. In the long-term that would have led to a healthier relationship and more substantive discussion.

I have to respectfully disagree with Dan on this. WOW was promoted as a book for a broad, non-academic audience. Berri even started a blog where he posts daily! He took his ideas into the public realm, and by his own choice. So a public discussion of his work seems entirely appropriate. He didn't object when non-academic Malcolm Gladwell wrote about his work in the non-peer-reviewed New Yorker; in fact, he talked about it all the time. Same with Matt Yglesias' blog commentary on WOW. Why? Could it be because he got mostly positive feedback there? So I think we know how to establish "a healthier relationship and more substantive discussion" with Dave: praise his work (in any venue you'd like).

But if you want to criticize the WOW, then the only legitimate venue is "a peer-reviewed refereed journal?" Sorry, I ain't buying. And isn't it convenient that this would give Berri a reprieve of many months before he had to respond to the paper? (Reminds me of President Bush's refusal to discuss the outing of Valerie Plame until all legal cases related to the issue are settled.)

The peer-review process is particularly unimportant in this case, as economists per se don't have any special expertise to bring to this debate. Mostly, this is a debate about the interpretation of sports performance statistics. Most economists, even sports economists, know far less about these issues than the non-academics who think and write about these issues. And to the extent this is a debate about the fine points of "econometrics," well, that's just a word economists made up to take ownership of statistical methods used in many fields. Many statisticians and people from other fields are qualified to comment on the legitimacy of Dan's statistical methods. Hopefully some will come forward and comment.

The paper is available NOW for Berri and his co-authors to review. They can enlist statistical experts to back up their criticisms (if they can find any). They can do their own analysis and try to show that WP does well at predicting future wins (something Berri has said he intends to do). If they believe in their work they will take up this challenge, rather than using the peer-review journal process as an excuse to hide.
Back to top
View user's profile Send private message
basketballvalue



Joined: 07 Mar 2006
Posts: 208


PostPosted: Thu Dec 13, 2007 1:18 pm Post subject: Reply with quote
Dan,

Thanks for publishing this paper. I'm looking forward to reading it in detail, I did enjoy the talk you gave in Boston at the conference and found it very insightful.

Also, I wanted to take a moment to thank Flint for being a part of our community and often representing the WOW position. It's unfortunate to me that the relationship between Dave and Dan is so negative, and perhaps someday they'll work it out. In the meantime, it seems to me that Flint is at least doing what many have asked of Dave, engaging us here on this board. Just because Dan and Dave don't get along doesn't mean that everyone who posts here should be mortal enemies with folks who post there and our paths should never cross. As I think I've heard others say in other threads, the more cross-pollination between the two communities, the better off we'll be. Perhaps we can persuade more folks who post there to join us here, and vice versa.

Thanks,
Aaron
_________________
http://www.basketballvalue.com
Follow on Twitter
Back to top
View user's profile Send private message
NickS



Joined: 30 Dec 2004
Posts: 384


PostPosted: Thu Dec 13, 2007 1:24 pm Post subject: Reply with quote
basketballvalue wrote:
Also, I wanted to take a moment to thank Flint for being a part of our community and often representing the WOW position. It's unfortunate to me that the relationship between Dave and Dan is so negative, and perhaps someday they'll work it out. In the meantime, it seems to me that Flint is at least doing what many have asked of Dave, engaging us here on this board. ... As I think I've heard others say in other threads, the more cross-pollination between the two communities, the better off we'll be. Perhaps we can persuade more folks who post there to join us here, and vice versa.


Thanks for posting this.

As someone who's posted quite a few comments on Wins Produced, my goal is always to have a discussion is an much depth as possible, not to be just lobbing attacks at a distant opponent.

This goes along with what Dan said earlier in the thread -- Wins Produced incorporates a number of interesting ideas; it's worth mucking around in the details, and the best way to do that is to have an interlocutor who is familiar with it.
Back to top
View user's profile Send private message
Westy



Joined: 15 Nov 2007
Posts: 15
Location: Chicago, IL

PostPosted: Thu Dec 13, 2007 1:48 pm Post subject: Re: Pot Kettle Black Reply with quote
I took the chance to read the paper on the train home last night. Very interesting. Not having attended the session in Boston, this was my first look at it. Also, not being an econometrician, much of this may be outside my realm of understanding, but nonetheless, I enjoyed it.

A few thoughts. First of all, very minor, but on the bottom of page 12 in the sentence that begins "The Appendix..." there appears there is a 'be' missing. Also, it might be nice if the formatting was slightly adjusted so you could print out the Appendix separately (start it on its own page, which it's close to being anyway).

Secondly, I think that a major aspect when it relates to advanced basketball statistics is the history. The paper notes that advanced baseball statistics seem to predict performance better than basketball based on this study. A major factor for this could be the relative ages of the 'movements'. This past year, SABR 37 was held. Has there yet been an official APBR convention? Bill James published his first Baseball Abstract in 1977. Dean Oliver's "Basketball on Paper" wasn't even published until 2004. Obviously there was work done before that, but the movement is much more recent than baseball. I think it would be well worth pointing this out in either the Introduction or the Conclusion of the paper. It's very relevant because it's worth noting that there may have been some point in the past where baseball statistics were unreliable, yet they have become moreso. It is very likely (I think anyway) that basketball statistics are on the same track. Yes, this study may show they're not there yet, but they will continue to evolve. Maybe basketball stats are way more complicated than baseball, but with future technology, new stats being added, and digital video, one would think we're only going to get better at this.
Back to top
View user's profile Send private message
cherokee_ACB



Joined: 22 Mar 2006
Posts: 157


PostPosted: Thu Dec 13, 2007 2:43 pm Post subject: Reply with quote
In chapter 6, why don't you also compute the correlation between the expected result and the actual result for each mini-game? That way, the analysis would not depend on the merits of adjusted +/- as a metric.
Back to top
View user's profile Send private message
Harold Almonte



Joined: 04 Aug 2006
Posts: 616


PostPosted: Thu Dec 13, 2007 3:18 pm Post subject: Reply with quote
I think the Berri's complain about D.R. playing the judge and a kind of accuser's advocate at the same time is a good and valid move, but the proposal of a blind comitee of economists revisionists with any relationship with sports metrics, or far from basketball game, is not. The methodologies is a secondary debate in all of this, the central point allways has been the hypothesis about the odds inside some basketball plays and its interpretation as a possession logic. I might be wrong, but not all kind of economist could manage that issue, if it were so, then all of this would have been solved the first weeks it began. And those who have been involucrated in the debate, immediately gets polarized inside one of the two schools.
Back to top
View user's profile Send private message
jamsesh



Joined: 12 Dec 2007
Posts: 6


PostPosted: Thu Dec 13, 2007 8:21 pm Post subject: Reply with quote
Dan, have you decided what your next move is? Are you going to play by Berri's rules or are you going to do things how you think they should be done?
Back to top
View user's profile Send private message AIM Address
Dan Rosenbaum



Joined: 03 Jan 2005
Posts: 541
Location: Greensboro, North Carolina

PostPosted: Thu Dec 13, 2007 8:44 pm Post subject: Reply with quote
jamsesh wrote:
Dan, have you decided what your next move is? Are you going to play by Berri's rules or are you going to do things how you think they should be done?

It soon will have went through just as much peer review as Wins Produced ever has.
Back to top
View user's profile Send private message Send e-mail Visit poster's website Yahoo Messenger
Dan Rosenbaum



Joined: 03 Jan 2005
Posts: 541
Location: Greensboro, North Carolina

PostPosted: Thu Dec 13, 2007 11:22 pm Post subject: Reply with quote
cherokee_ACB wrote:
In chapter 6, why don't you also compute the correlation between the expected result and the actual result for each mini-game? That way, the analysis would not depend on the merits of adjusted +/- as a metric.

That is a good suggestion, but the result would be a lot of really, really low correlations. Who wins those mini-games is mostly luck. The low correlations are not an econometrics problem - because I suspect the power would be even greater with this analysis. But the low correlations would make it easy for folks with shaky econometrics understanding to dismiss the analysis. Given the lack of understanding about the noisy dependent variable issue (by Berri among others), I worry that these low correlations would also be hard to sell. It may be something that we end up doing, but I am not confident it will help as much as it should.
Back to top
View user's profile Send private message Send e-mail Visit poster's website Yahoo Messenger
Charles



Joined: 16 May 2005
Posts: 134


PostPosted: Fri Dec 14, 2007 2:31 am Post subject: Reply with quote
I found it a thorough and interesting technical analysis.

The theories of why the Wins Produced variants did so much better on predicting future season wins than explaining adjusted plus/minus seems critical. Let me offer a further possibility.

It seems that the biggest problem with Wins Produced is the over valuing of individual rebounding. Furthermore, it seems likely that the issue is more pronounced when comparing players in different roles than when comparing players who play similar roles for different teams.

If that is correct, then it follows that it may be difficult to pick up the over-emphasis on rebounding when using current data to predict a future season's results. When you compare year-to-year every team will likely have a similar rebounding system. So, for example, if a team exchanges one over-rated rebounder for another over-rated rebounder, who's to know?

However, adjusted plus/minus deals with all kinds of lineup structures. Everything from three guards and no center to one guard and three players who could be classed as bigs. And, of course, everthing in between. This makes it difficult for team or position adjustments to disguise the underlying problem with rebounding. For example, when one team goes small, Wins Produced may tend to predict the "bigger" team to dominate. When that doesn't occur, the correlation will drop.
Back to top
View user's profile Send private message
Dan Rosenbaum



Joined: 03 Jan 2005
Posts: 541
Location: Greensboro, North Carolina

PostPosted: Fri Dec 14, 2007 8:40 am Post subject: Reply with quote
Charles wrote:
The theories of why the Wins Produced variants did so much better on predicting future season wins than explaining adjusted plus/minus seems critical. Let me offer a further possibility.

I am not quite sure what you are talking about here. The correlations are lower with adjusted plus/minus than with team wins, but that is to be expected. Correlations and R^2 are typically higher for aggreagated data, which is what we are looking at in the team wins analyses. And Wins Produced was among the worse predictors in both the team wins and adjusted plus/minus analyses, so I am a little confused as to what you are saying there.
Back to top
View user's profile Send private message Send e-mail Visit poster's website Yahoo Messenger
Eli W



Joined: 01 Feb 2005
Posts: 402


PostPosted: Fri Dec 14, 2007 9:00 am Post subject: Reply with quote
One suggestion I have is to run the team wins and adjusted plus/minus tests on all the rating systems without any team adjustments included. And running the tests with all the ratings having the exact same team adjustment that Wins Produced has.
_________________
Eli W. (formerly John Quincy)
CountTheBasket.com
Back to top
View user's profile Send private message Send e-mail Visit poster's website
kjb



Joined: 03 Jan 2005
Posts: 865
Location: Washington, DC

PostPosted: Fri Dec 14, 2007 11:04 am Post subject: Reply with quote
The problem with running the tests with each rating having the exact same team adjustment Wins Produced uses is that the WP adjustment is tailored to WP. I don't think it would work. Maybe Dan could make it work by using the same factors, but applying differing values.

Before I knew how the WP adjustment worked, I used a very simple defensive adjustment for players. It was basically the percentage difference between team defensive rating and the league average. So, if the league average efficiency was 106 and a team drtg was 101, players for that team would get roughly a 5% bonus on their individual stats (as summarized by the rating systems I was looking at (Win Score vs. Efficiency vs. PER vs. a TENDEX-like system I'd been using).

With that defensive adjustment, I found that at the team level, ALL of the rating systems did a good job of describing same-season wins. As I recall, they all had correlations above .94. PER came out #1, but it was a very minor difference with the other rating systems.

I want to emphasize that the defensive adjustment I used is not close to how Berri does it in Wins Produced.
Back to top
View user's profile Send private message AIM Address Yahoo Messenger
Charles



Joined: 16 May 2005
Posts: 134


PostPosted: Fri Dec 14, 2007 11:30 am Post subject: Reply with quote
Dan Rosenbaum wrote:
Charles wrote:
The theories of why the Wins Produced variants did so much better on predicting future season wins than explaining adjusted plus/minus seems critical.

I am not quite sure what you are talking about here. The correlations are lower with adjusted plus/minus than with team wins, but that is to be expected. Correlations and R^2 are typically higher for aggreagated data, which is what we are looking at in the team wins analyses.

I was referring to the correlations of the WoW methods relative to the other metrics. Is that not what you meant when you pointed out that “Alternate Win Score – performs the best predicting team wins, but does not perform as well explaining adjusted plus/minus.”

Perhaps, it was not clear that I was thinking of AWS as a variant of WP.

Dan Rosenbaum wrote:
Wins Produced was among the worse predictors in both the team wins and adjusted plus/minus analyses, so I am a little confused as to what you are saying there.

Yes, WP trails in both measures, but don't you think that the rebounding element in Wins Produced is so problematic that it will cause it to perform poorly as a predictor of any reasonable barometer? The point here is that it is much further behind in adjusted plus/minus.

All that aside, does it make sense to you that WP’s [over]-emphasis on rebounding may be somewhat counterfeited when predicting year-to-year wins, but more clearly exposed by plus/minus? That seems like a major difference in the two barometers.


Last edited by Crow on Thu May 12, 2011 6:54 am, edited 1 time in total.

Top
 Profile  
 
PostPosted: Fri Apr 15, 2011 6:48 am 
Offline

Joined: Thu Apr 14, 2011 11:10 pm
Posts: 2683
page 3 of 3

Author Message Harold Almonte



Joined: 04 Aug 2006
Posts: 616
Posted: Fri Dec 14, 2007 12:14 pm Post subject:

I will dare to make this argument, even not being a statician (sorry), and not understanding too much about regressions and residuals. It could be said that the basketball game is a game where there are more residuals that are not boxscored than boxscored variables (specially at defense), and probably some of these residuals are as worth as a high percent of its attached variable in some plays, and probably you can make up for all of that. If you do that make up in a fair way could be secondary and could be not. But the odds. We know that scoring depends on odds. A FGA can be made or missed and the odds produce a break even that rate the action, but then we need to weight the two parts of the action, and suddenly we realize that "mades" depends on the scoring prize and sometimes on teammates assist help, but "missed" itself (for the team of course), depends on a new odd, the ball to be rebounded by your team or by the opponent, wich at the same time is an odd for the act of rebounding (between opponents). My low level statician mind can't understand why odds between teams can't weight individual variables (FGMissed in this case) and a inter-team action (Rebounding from FGMissed, or from blocking).
Back to top

Eli W



Joined: 01 Feb 2005
Posts: 401
Posted: Fri Dec 14, 2007 2:42 pm Post subject:

kjb wrote:
The problem with running the tests with each rating having the exact same team adjustment Wins Produced uses is that the WP adjustment is tailored to WP. I don't think it would work. Maybe Dan could make it work by using the same factors, but applying differing values.
I don't think this would be a big problem. If you read the section titled "Step Three" at the following link you can see the Wins Produced team adjustment (though note that the example leaves out 3PM because it's pre-three-point line). http://www.wagesofwins.com/CalculatingWinsProduced.html It seems like this could be applied to PER or NBA Efficiency with just some slight tweaks._________________Eli W. (formerly John Quincy) CountTheBasket.com
Back to top

Harold Almonte



Joined: 04 Aug 2006
Posts: 616
Posted: Fri Dec 14, 2007 2:53 pm Post subject:

The team defense adjustment is just to account opp. team points, opp. FGMade, and other opp. team possessions, and distribute all of this to the players in a team, by minutes. It's not such a complicated thing and there's no room for DR to change anything if all is taken from the same boxscore. Basically it's just penalizing for points allowed, and creating the holy grail-ed-points differential thats correlates so high with wins. The quid is that all rating that points differential is added, will correlate well with wins (as well as not scoring possessions are well weighted). Points allowed can be distributed arbitrarily and doesn't change the team wins correlation, but the rating of players becomes a disorder. Since you take account just minutes and nothing more to distribute, then low usage-high minutes players gets overrated.Last edited by Harold Almonte on Fri Dec 14, 2007 4:34 pm; edited 1 time in total
Back to top

Guy



Joined: 02 May 2007
Posts: 128
Posted: Fri Dec 14, 2007 4:11 pm Post subject:

Eli: I raised this idea with Dan myself, but the team adjustments really do have to be quite different depending on the metric. That's especially true for MP and PTS, which use very different information in the core metric. It might be interesting, as a side project, to create team adjustments just for AWP and PER using the same component variables as Berri's adjustment. But the only purpose of that is to show Dave B. that those metrics still out-predict WP w/o recourse to the dreaded "residuals." I don't think Dan and Dave should change the paper just for that purpose. And dispensing with team adjustments entirely will just weaken the predictive power of all the metrics; again, not something that should be done for the paper (in my view).
Back to top

Harold Almonte



Joined: 04 Aug 2006
Posts: 616
Posted: Fri Dec 14, 2007 4:24 pm Post subject:

I don't understand how it can vary the information about MP and Points. But what I'm wondering is how the assisted factor was canceled in the PER team defense adjustment. I would like to see also a win correlation of PER-counterpart PER


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 2 posts ] 

All times are UTC


Who is online

Users browsing this forum: Google [Bot] and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group