Referreed Journals

[mystic]

I'm not sure at your confusion

Sorry, but you are the one being confused here. Evan NEVER said anything about "not allowed to comment" at all. Berri can comment as much as he likes, but that doesn't make him a a subject matter expert in any of those things except of economy. And that's what Evan said. You made his point by citing other economist who aren't subject matter experts, you even said that economists aren't subject matter experts with the exception of economy. But when someone wants to comment about a subject he is not an expert in, he might as well listen to subject matter experts.

Berri is not well educated in basketball stats, he knows his metric, he knows NBA Eff and PER. That is basically all he knows about that. Berri also shows that he doesn#t understand that correlation does not mean causation. When he started he didn't even include assists as part of his WP metric, because his simple linear correlation analysis didn't show an effect. It took years before he included assists while defending his metric as superior to everything. And somehow he did include assists later, despite saying before that assists wouldn't matter. The next was the rebounding issue, with wasn't an issue to Berri at all. He dismissed critiques on that part with some other simple linear correlation he has found. And finally after years of discussion and with a lot of name calling by Berri he changed the rebounding part. Funny thing is that his new WP is better at predicting than his old WP, but the new WP correlates less from year to year than the old one. He traded reliability for validity, but not on his own based research, but after A LOT of people told him that.

That Berri was able to publish his stuff in journals rather proves that the amount of real experts in that field who are working in the academic world is really low, not that Berri is an expert. And given the fact that Berri isn't capable of including financial restrictions caused by the CBA in his analyses is also quite disturbing. Calling decision makers in the NBA "dumb" while not being able to include another layer which has to be taken into account when making a decision, is rather funny. Last season for example he called out the decision makers of the Hornets, because a trade did not improve the Hornets according to WP48. But the financial situation of the Hornets became better, they traded players with longer contracts for shorter contracts to improve financial flexibility, which is a huge thing in terms of building a better team in the future. No idea, but that whole article sounded like it was written by someone who has never seen the term salary cap in his whole life.

Really? That's been shown? I'd love to see that demonstrated. Can you point to an example or article or something similar?

Yes, it was. Multiple times already. Rosenbaum showed in 2007 for example. And this thread there is even a link for this.

I checked the ability to explain lineup performances via different metrics, and WP48 was as good as PER, while being clearly worse than WS/48 and my own metric. I only used the Top50 lineups in each year in minutes played in order to get a better sample size. In essence, you can roll a dice or use WP48 in order to make a lineup decision, it has basically the same chance of success.

That's not true. They said you shouldn't be judged on what you DIDN'T know.

Nobody wants to do that. But that isn't the point as Guy pointed out. WoW is only using reliabillity, but not validity. The validity is assumed, because it is correlation there, but it was never shown that WP48 is a valid player evaluation tool. In fact I needed like 10 minutes to show that a system based on scoring per 100 possession + defensive adjustments has a slightly better explanatory power and slightly better reliability than WP48. That system was based on a linear regression. It fullfilled all the requirements Berri said are necessary in order to "prove" it is a "good" metric. And it is more simple than WP48. Well, everyone with only a small amount of knowledge to the subject will say that FM48 (that's how I called it) is missing things. But well, as I said, not according to Berri's guideline. The issue is that Berri's guideline is not telling you much about the quality of the model. If someone would try to convince the physics community that his model is the correct one and he would use Berri's guideline, he wouldn't have a chance. Now, go back to WOW and ask whether they are willing to prove that the model is valid by using out-of-sample tests, that there are willing to show that a lineup with better WP48 is indeed performing better (at least on a somewhat consistent basis). When other boxscore based metrics are showing that a correlation coefficient between the average of a specific 5-man-unit and the result can be over 0.67, WP48 should be able to do the same, if it indeed evaluates players.

[Guy]

Really? That's been shown? I'd love to see that demonstrated. Can you point to an example or article or something similar?

Um, look at the link in the post immediately before yours, where Sport Skeptic demonstrates this. See also the comment there by Neil Paine, who has recently found the same thing using a different methodology.

Anyway, if we exclude people without specific domain knowledge from commenting on a topic, we'll be left with what?

You raise some valid questions above about who should be considered a domain expert, and I agree that "expert" should not be defined too narrowly. Nor should we simply defer to "experts." But the importance of domain expertise varies depending on the question being explored, as Levitt's work shows. Levitt's abortion/crime analysis was basically an exercise in demography, and his skill set I think qualifies him to explore a possible relationship between abortion and crime rates w/o deep expertise in criminology (or obstetrics). (Although this study is not w/o critics.) However, in the second Freakonomics book he ventured deeped into questions that do require deep domain knowledge -- e.g. global warming -- and as a result his work was widely criticized.

We can debate how much domain knowledge is required to measure basketball performance. But we know that domain experts told Berri from the beginning he was overvaluing rebounds, but he didn't listen, only to concede the point (sort of) six years later. Domain experts also told him there is a usage-efficiency tradeoff, which means WP is systematically undervaluing high-volume shooters. Again he ignored them, and again he was wrong (perhaps by 2018 he will fix this problem?).

In any case, you set up a straw man here. Evan didn't suggest that non-experts be "excluded" from the discussion. That would be odd indeed, since he concedes he's not a domain expert himself and yet clearly wants to be part of the discussion. What Evan said is "Where I am told by someone who I trust as a basketball expert that certain stats are insufficient, I try to weigh that criticism heavily." Not "accept blindly," but "weight heavily." Good advice, and advice that Berri should have heeded.

[EvanZ]

I think we can agree that Levitt is no more a domain expert in crime than Berri is in basketball, no?

As said, you'd be hard pressed to find an economist who is a domain expert in something they have applied economic and statistical analysis to. is a great talk on the effects of various programs to reduce AIDS, a GREAT watch BTW. I highly recommend it.

It's not called the "dismal science" for nothing.

The idea that Berri continually seems to promote is that all the experts are wrong (about pretty much everything) and that his metric proves it. And I honestly don't think he would disagree with what I just said. Don't you find that odd?

[mystic]

The idea that Berri continually seems to promote is that all the experts are wrong (about pretty much everything) and that his metric proves it. And I honestly don't think he would disagree with what I just said. Don't you find that odd?

Not only this. In his book "Wages of Wins" for example, Berri pretty much says that the stuff Roland Beech is doing is unnecessary and doesn't help. Beech sat on the bench of the Dallas Mavericks when they won the championship. What is Berri doing? Claiming that the Mavericks would use WP48 secretly in order to make decisions.
Let alone that in his book he claims that Winston-Sagarin ratings wouldn't be much else than the stuff Beech done. That shows the point I made about him being not that knowledgable. In 2004 was a long article published in the Washington Times about Winston-Sagarin and adjusted+/-, every half-smart reader of that article should have understand that this is about regression and has not much to do with unadjusted +/- like Beech is using it in his Roland Ratings. Two weeks later Rosenbaum published his article about APM on 82games.com, in which the algorithm is explained. The part of the APM is shown. Well, two years later Berri still hasn't gotten the memo, he implies in his book that Winston-Sagarin would be unadjusted +/-.

[EvanZ]

To my knowledge, he still hasn't acknowledged that RAPM even exists.

[LA Blue Devil]

Regarding Berri and his publishing record, I have some information that I think a lot of you guys will find interesting. I have read a number of Berri articles "cover to cover" because I am using a number of them in my literature review. I won't get into what I think about the content of Berri's research because there are other threads for that, but a couple of interesting trends emerged. When I searched the EconLit database, it returned 36 journal articles that were authored or co-authored by Berri (He says on the WoW website that he has 43 publications and this is not meant to refute that, as EconLit is sometimes missing international and lesser known journals), and this is some of what I found:

-Of the 36 articles listed, 16 were published in the 2 journals in which Berri is on the editorial board
-12 of his articles were published in the Journal of Sports Economics (JSE), which is an e-journal "published in conjunction with" the North American Association of Sports Economics (NAASE), of which Berri used to be President
-According to the NAASE website, anyone (academic or not) is able to join as long as they pay the $20 dues a year
-Berri has only published one article in a major economics journal (American Economic Review) and the paper had nothing to do with WP. (It was his article about the effect of lockouts on fan interest and attendance)
-JSE has an Impact Factor (IF) of 0.528, which ranks 199 out of 305 journals in economics and 29 our of 33 in its category of "Hospitality, Leisure, Sport & Tourism" (Impact Factor is essentially a proxy measure of a journal's importance by looking at the number and quality of its citations. It is by no means perfect, but it is the standard in academia)
-Outside of his article in the American Economic Review, all of the 36 journals ranked in the bottom half of economics journals in Impact Factor

It may seem like I am trying to "attack" Berri and not his work, but this is his work! As Berri says on the WoW website, "This advice follows from the incentives facing academics. Professors are not just asked to teach college courses. We are also expected to do research, and this research is supposed to be reviewed by our peers. Research that passes peer-review is then published, and it is the quantity and quality of our publications that are primarily used to evaluate our performance (just for the record – as of January, 2011 – Martin Schmidt has 35 publications in academic journals and academic collections while David Berri has 43 publications in journals and collections)." While he was sure to point out the quantity side, he is silent on the quality side. I get the impression that Berri think that his PhD means that his work is beyond the criticism of anyone without one.

[mystic]

To my knowledge, he still hasn't acknowledged that RAPM even exists.

Yeah, that's my impression as well. And Joe Sill made his website back in 2009, he showed the algorithm, he showed the numbers. Reading his explanations and knowing how regularizations works, I immediately started citing RAPM instead of APM (Sill had the numbers from 2006-07 to 2009-10 on his hoopnumbers.com).

http://forums.realgm.com/boards/viewtop ... #p22576939
That is a link to post in which I argue with RAPM by Sill back in March 2010! And two years later Berri is still arguing about unadjusted +/-, doesn't have fully understand the APM by Barzilai or Iliardi and is far away from understanding RAPM. What kind of expert is that?

@LA Blue Devil

I done a similar thing when I got his book and notice the same (well, not to the extend that I checked the editorial boards of the journals). His "work" isn't really valuable, his methods are simple (most times simple linear correlation analysis or linear regressions), not often cited and published in minor journals (as far as I could tell as a non-expert in economics). That's why I even doubted his knowledge in economics (a couple of times on RealGM). Berri isn't a great researcher, maybe he is a better teacher?

[EvanZ]

-JSE has an Impact Factor (IF) of 0.528, which ranks 199 out of 305 journals in economics and 29 our of 33 in its category of "Hospitality, Leisure, Sport & Tourism" (Impact Factor is essentially a proxy measure of a journal's importance by looking at the number and quality of its citations. It is by no means perfect, but it is the standard in academia)

That's quite low. To put this in perspective, my last published paper was in a journal called Developmental Dynamics that has an impact factor of around 3.0. Before that, I had papers in PLoS Biology (~12) and PNAS (~9). I think the lowest impact factor for any of my publications was around 1.5.

The goal for any scientist is to get something in Science or Nature which have IF around 30 or so, if I recall correctly.

[LA Blue Devil]

Mystic: This is completely a guess, but I highly doubt he is a very good teacher just based off of the way that he treats people who he considers "less than him." I got the editorial board information from his CV, and based off of the classes that he listed, he has only taught one class that has anything to do with sports. As for his credentials as a researcher, I can't speak to his knowledge of economics, but my thesis adviser can. He is a "big name" in the econometrics field who happens to be a basketball nut, and when I showed him some of Berri's work, he was pretty surprised that some of it had been published by an actual peer review journal. He was actually the one who had me start to look into the publications that are publishing him.

EvanZ: I didn't want to get into too much detail on the IF stuff before, but you might find this interesting. Other than American Economic Review, none of the journals had an IF that topped one, and six of them don't top 0.6 (These include the two journals that he published 16 of the 36 in). Also the Thompson Reuters database that I'm using lists an Adj. IF that adjusts for self-citation, under which Journal of Sports Economics has an IF of 0.292. Again not a direct critique of WP, but I think this is pretty revealing about his quality as a researcher

[Guy]

I don't think the prestige level of Berri's journals tells us anything about the quality of his basketball analysis. I don't believe Berri's work deserves any special deference because of his PhD or the peer review process. But then let's be fair: critics can't invoke his academic pedigree (or lack thereof) only when it suits their purposes.

The fact is that Berri knows more about basketball than many sports economists know about the sports they write about. I've read journal articles that miscalculate home field advantage in baseball because the author didn't know that home teams don't bat in the 9th inning if they are already ahead! And if anything, I'd guess the relationship between substantive sports knowledge and academic prestige is negative. For example, my sense is that Ray Fair at Yale is incredibly well-respected in the economics field, but he wrote a positively dreadful paper in 2008 on the aging curve for baseball players. If Berri were one of the country's top economists, it wouldn't make any of think that WP was a better metric; so let's not pretend that a lack of academic prestige (even if true) tells us anything either.

[mystic]

I don't think the prestige level of Berri's journals tells us anything about the quality of his basketball analysis.

Indeed, it doesn't.

The fact is that Berri knows more about basketball than many sports economists know about the sports they write about.

Really? That is quite sad.

If Berri were one of the country's top economists, it wouldn't make any of think that WP was a better metric; so let's not pretend that a lack of academic prestige (even if true) tells us anything either.

You are right, going over his published papers doesn't change anything about WP. I guess, it is just weird to see someone making such a big fuss about himself and his academic work, when in reality it holds not much water.


But we can use Berri as an example here. It is rather obvious that the online community in basketball stats is much faster in terms of spreading knowledge. We have published articles in 2004 about APM, and Berri had no clue that this exists in 2006. It was a big thing when Joe Sill made his stuff about regularization and a lot of people saw the advantages in that, while Berri and his academic world seems to not know that Tikhonov regularization even exists, as of right now. So, how much value does have the peer-reviewed papers in such a context?

[mtamada]

Also the Thompson Reuters database that I'm using lists an Adj. IF that adjusts for self-citation, under which Journal of Sports Economics has an IF of 0.292.

Wow, I haven't looked into Impact Factor in detail, I'd always assumed that one of the first things it did was throw out, or at least put a lower weight on, self-citations. It's sort of the flip side of an initial design of box score stats: to fully include self-cites is as naive as counting all field goals while not counting assists. It's better than no box score at all, but it's a naive, primitive way to count.

It is true that when a field is new and small, there is less literature to cite and sometimes the relevant citation is simply one's own previous work. But sports economics, though relatively new and small compared to say labor economics, isn't that new and small.

[EvanZ]

Also the Thompson Reuters database that I'm using lists an Adj. IF that adjusts for self-citation, under which Journal of Sports Economics has an IF of 0.292.

Wow, I haven't looked into Impact Factor in detail, I'd always assumed that one of the first things it did was throw out, or at least put a lower weight on, self-citations. It's sort of the flip side of an initial design of box score stats: to fully include self-cites is as naive as counting all field goals while not counting assists. It's better than no box score at all, but it's a naive, primitive way to count.

It's probably not a major factor for high IF journals. While you may cite your own article regardless of where it is published, if it is published in Science or Nature, it will be published much more often by others.

Eigenfactor is a newer metric that is probably an improvement, but much less often used:

http://www.eigenfactor.org/methods.php

[mtamada]

I don't think the prestige level of Berri's journals tells us anything about the quality of his basketball analysis. I don't believe Berri's work deserves any special deference because of his PhD or the peer review process. But then let's be fair: critics can't invoke his academic pedigree (or lack thereof) only when it suits their purposes.

[...]

my sense is that Ray Fair at Yale is incredibly well-respected in the economics field, but he wrote a positively dreadful paper in 2008 on the aging curve for baseball players.

Yeah, Fair's a top econometrician. Not Nobel-level top, but a name instantly recognizable to any economist. And yeah that paper's not great although I don't think I'd go so far as to say positively dreadful. He correctly notes in the paper that his model is highly parsimonioius -- a parametric model with basically just three parameters (plus one additional constant for each player). It's a classic econometric tradeoff though against models which permit greater flexibility (by having more parameters or by being semi-parametric), but which have less reliable estimates.

Your first point of course is correct: when it comes to sports analytics, the academic standing of the author is of minor concern. Nonetheless it's worthwhile to take a look at Berri's background given that he tends to justify his claims and downgrade others' based on academic background. Berri's is assuredly not an instantly recognizable name amongst economists.

As for his work, as opposed to his academic background: I haven't paid much attention to him since he brought his initial work to the old APBR group, using the Cobb-Douglas production function to estimate NBA productivity. Having an economics background myself I was glad to see someone attempting to apply econometrics to basketball. Two immediate problems though: he took the coefficients from his team regression and assumed that they could be applied to individual players. And when he proclaimed that his results showed that Dennis Rodman was the best player in the NBA, and commenters objected (that's when APBR first used the term "laugh test") ... well I think the econometrician Henri Theil put it best: "Models are to be used, not believed". Berri hangs on far too grimly to his results. In contrast, JohnH knows that PER is just an index, not a true measure of player quality; we're all aware of the limitations of APM, RAPM, etc. and continue to fiddle with them (Wayne Winston however like Berri seems unable to notice the limitations of his model); we compare different models with the notion of taking the best ideas from them; etc. etc. (Which is also the way that economics and inded all academic fields are supposed to work, and usually do work -- but with notable exceptions.)

[Crow]

A couple days ago Berri called Milsap the best PF in the NBA based on WP. Ryan Anderson was tied for second.

I checked non-prior informed RAPM and the top PFs on that are T Young, Gibson then Anderson with a slight over over Milsap and Harrington.

Those 2 player ratings are pretty much in agreement.

But there are differences with others. Love is tied for second on WP but barely above neutral on non-prior informed RAPM. He has always been higher rated on WP than RAPM.

Ibaka 7th best on WP but barely above neutral on non-prior informed RAPM.

WP doesn't think that much of Bosh but non-prior informed RAPM has him as 7th best.

Most of WP's top 10 are not rated highly on non-prior informed RAPM.

Neither metric rates Dirk, Blake or KG highly.

It is early for both WP and non-prior informed RAPM so I woudn't make too much of either.

Some of cases where the 2 metrics diverge noticeably may say something about those players and specifically their non-boxscore dimension. Some of it may be noise in RAPM or noise in personal stats. The divergance at least raises questions for me that might not come up if I relied on just one metric.