Referreed Journals

[royce.toyfu]

I know this is probably fighting the last war, but Krugman had a blog entry yesterday talking about how referreed journals have been gamed for decades and that the new online discussion model is preferrable.

http://krugman.blogs.nytimes.com/2012/0 ... ogosphere/

[Ryan]

It's kind of like the internet. Just because it's published in a journal doesn't mean it's right.

That issue aside, I've found journals to act as a nice archive. Can you imagine trying to find stuff people posted on blogs and other websites in say 20 years from now?

Maybe someone should create journal blogs, which I will forever refer to as jogs.

[EvanZ]

As an academic myself, my opinion is that it would be very difficult to change to this system. We live in a bean counting world (for tenure, grants, etc.). Nobody even cares about conferences anymore. It's all about what high impact journals you're publishing in.

[royce.toyfu]

I find dichotomy of Nobel Laureate, John Bates Clark Medal Winning Paul Krugman's, and Berri's opinions on online discussion and refereed journals interesting. Krugman could be the ultimate arguer from authority, yet engages online authors with interesting ideas, Berri dismisses anything not in a refereed journal he deems worthy of considering.

[EvanZ]

Berri uses it as a litmus test simply because it's expedient for him to dismiss the majority of his critics who are obviously not publishing APBR article in peer-reviewed journals (or any "journals" for that matter).

[xkonk]

This is obviously an area that can only create arguments, but I think there are points to both sides. In my area of academia, there are no pre-published papers like Krugman describes. Authors might get feedback from other people in their department or occasionally from other researchers working on similar projects, and there are many conferences where you can find out about what research is going on, but published papers are far and away the main method of conveying results. Indiscriminately calling the journal system 'broken' seems overly strong to me.

Academic work also benefits from a certain amount of barrier to entry and the publishing system. To publish something, you almost always have to be in a university program. That means you've taken university classes and have a mentor; you have some amount of training. Papers often have multiple authors, meaning that the data has probably been vetted by more than one person, and the claims made in the paper have been vetted (hopefully) by everyone whose name is on the paper. The peer review process ensures that another group of people have looked over the paper and not seen anything terribly wrong (again, hopefully). So when you read an academic article, you can feel fairly certain that the content has been found accurate and reasonable by at least five or six experts on that topic, if not more, and the work was done by people that have directly relevant training. Also, that ignores any other suggestions or fixes that happened if the research was presented at talks or conferences before being written up; a good number of people have contributed to most published articles. Of course, as Phil Birnbaum will tell you, academic research has mistakes. The process isn't perfect.

On the other hand, anyone can post something on their blog. Analyses are usually done by one person on data that may or may not be well-described. The person doing the analyses may or may not have any training in programming or statistics; even if they get Excel or R or whatever to run a test on the correct data they may not know if it's a proper test or the best test or what aspects of the output are important. It's highly unlikely you really know the person doing the work, or anyone who has worked with that person. Results are often just thrown up on the internet with a certain lack of detail, because academics expect to read 10 page papers but people online do not. So there is little barrier to entry, there is little assurance of researcher ability or qualification, there is virtually no oversight, and these issues apply to the readers as much as the writers. Now to go the other way again, that is not to say that everything written online is wrong or written by unqualified people. Obviously a number of people in the community have statistical savvy and should well be considered experts in the field. But I do think that if you were to take any random blog post and compare it to any random journal article, it's fair to assume that much more quality work went into the article than the blog post.

Berri takes a lot of flak in this area; people don't seem to like him or Wins Produced. If you look through some of the comments that used to come up on his site, I don't blame him for being a jerk to some of those people. But he has work and opinions on things unrelated to Wins Produced, and I haven't seen any big complaints about those. Presumably not everything he says is wrong.

[royce.toyfu]

Well, if you want to compare vitriol in online comments, I'm pretty sure Krugman wins over Berri hands down. He still manages to not dismiss all of it as beneath him.

And yes, anyone can post something on their blog. Generally it will get ignored. If someone else looks at it and finds it compelling, they may help spread it, or respond to it. If it's compelling enough lots of people will read it, and generally critique it. If it's just voodoo it will go back to being ignored. If Berri hadn't been able to skip straight to Book form and had just posted his stuff on a blog it may have been largely ignored, the refereed journal aspect of it lent it more credibility than it may have deserved.

[EvanZ]

I think the difference here is that academics are usually domain experts. I wouldn't argue with Berri's expertise in economics. I can only assume he's an expert, given his academic credentials. But he's not a basketball domain expert. And that's the thing that rubs people the wrong way. He assumes that there is an equivalency between "stats expert" (or econometrics expert) and basketball domain expert. That is simply not the case. And his analysis would benefit from acknowledging that limitation and working on improving it. I know in my own personal experience, I started coming at this basketball stuff from the same perspective, purely looking at the stats. But I have gradually realized the limitations in that approach, and I am trying to improve my knowledge of the game. Where I am told by someone who I trust as a basketball expert that certain stats are insufficient, I try to weigh that criticism heavily.

[motherwell]

I think the difference here is that academics are usually domain experts.

I don't think that;s true. Steven Levitt isn't a "domain expert" in either crime or abortion, to use perhaps the most popularly famous economics example. Emily Oster (TED talk: isn't a domain expert on medicine or AIDS.

In fact, you'd be hard pressed to find a single economist that has an opinion on a topic outside direct economic theory that is a domain expert in anything EXCEPT economics.

The question is does that matter? Can stats tell us something, and if so, what and how? If not, why does anyone bother?

And his analysis would benefit from acknowledging that limitation and working on improving it.

Can you give me an example? I am sure this isn't a straw man argument, but without quoting an analysis, that is what it amounts to. Is there a specific analysis that Berri has made that demonstrates this issue?

Where I am told by someone who I trust as a basketball expert that certain stats are insufficient, I try to weigh that criticism heavily

Insufficient for what purpose? This is actually a very interesting question. "Insufficient" can only relate to decision making, and would mean that whatever is being proposed misses something that significantly affects the ability of the observation to be believed.

As an analogy, I would believe an expert on Jewish history telling me that my population count was short because I missed a few communities he knew of, but to say "you are missing a few communities I am sure" without knowing where, I'd be a bit more skeptical.

Can you give me an example of a observation that makes a metric "insufficient" in making a decision?

[mystic]

In fact, you'd be hard pressed to find a single economist that has an opinion on a topic outside direct economic theory that is a domain expert in anything EXCEPT economics.

No idea what you have read in Evan's post, but you made his point. Evan said that academics are likely experts in their respective domains, but not necessary outside of their domains. Berri might be an expert in economics, but he is hardly an expert in basketball (or basketball stats for that matter).

Insufficient for what purpose?

Predictions. WP48 isn't good at predicting future outcome of games, it isn't good in out-of-sample tests. Getting the amounts of wins right after all games are played out is an easy task, but being able to predict depends on the validity of a model as player evaluation tool. WP48 fails in such tests.

This is actually a very interesting question. "Insufficient" can only relate to decision making, and would mean that whatever is being proposed misses something that significantly affects the ability of the observation to be believed.

Well, obviously WP48 is missing something. That something is called "interaction between players". WP48 handles players as if each boxscore entry would be result of an isolated event. That is just not the case. That is the very reason why WP48 isn't good at predicting, it is just a bad player evaluation tool. In hindsight saying that a team with a certain scoring margin would win a certain amount of games isn't some special knowledge Berri discovered.

Can you give me an example of a observation that makes a metric "insufficient" in making a decision?

Being a bad tool at predicting should be example enough. Just ask Berri&Co. what they think about out-of-sample tests and why it is not a good way to test a model. You either believe that a model doesn't need to be good at predicting or you are smart enough to realize that WP48 isn't a good tool for player evaluation.

[EvanZ]

WoW had an article the other day that said prediction doesn't matter. Nuff said.

[xkonk]

WoW had an article the other day that said prediction doesn't matter. Nuff said.

I missed that article. Do you mean this one? http://wagesofwins.com/2011/12/23/how-t ... edictions/ . It seems like the main issue they argue is that Wins Produced shouldn't be judged solely on the outcome of one season's predictions that also included a lot of extraneous factors, like different minutes, predictions, etc. In the predictions thread didn't we all just agree that was perfectly reasonable?

I guess I'm also curious as to what makes someone a 'subject matter expert'. Is it work in the field? I'm sure Berri meets that qualification. Opinion of other people? Which people? It doesn't seem obvious that dislike for a person should disqualify them from being an expert. How many people that post here are basketball experts? Why? Unless I'm mistaken, this is the same group that produced and supported APM, and that turned out to be a mistake. So developing and using a bad metric can't be the only strike against expertise, right? Is Henry Abbot at True Hoop a basketball expert? Watching, analyzing, and talking about basketball is his job, but on stats issues I would say he's probably wrong about as often as he's right. But, virtually regardless of what he says the majority of comments are against him. Can we establish some guidelines?

[J.E.]

this is the same group that produced and supported APM, and that turned out to be a mistake

It can only be considered a mistake if those people believed that APM was a good metric regardless of the amount of data. Multi year APM is a good metric

[Guy]

xkonk: I agree that your linked article raises many reasonable objections to using any single year's outcomes to judge a metric. However, the larger context here is that some in the WOW community used to be interested in predictive power (properly done) as a valid way of evaluating metrics, but then seemed to move away from that position as evidence accumulated that WP did not predict future wins as well as other metrics. While Dre does not quite come out and say that "predicition doesn't matter," he does seem to suggest that predicting future performance is just a "fun" exercise with no real analytic importance.

But perhaps I am misinterpreting Dre. Conveniently, we now have a good test: friend-of-WOW Sport Skeptic has just compared the predictive power of various metrics here: http://sportskeptic.wordpress.com/2012/ ... the-goods/. None of Dre's objections apply to this exercise. As you'll see, WP fares rather poorly, doing worse than both the best version of plus/minus and also two boxscore metrics (Win Shares and Daniel's ASPM). To me it seems very hard to square these results (along with Dan Rosenbaum's earlier and similar results) with the idea that WP is correctly weighting boxscore statistics. So it will be interesting to see how -- and whether -- the WOW community deals with Alex's findings.

[motherwell]

No idea what you have read in Evan's post ... Berri might be an expert in economics, but he is hardly an expert in basketball (or basketball stats for that matter).

I'm not sure at your confusion, because I actually quoted what I was responding to. But my apologies for not being clear. http://en.wikipedia.org/wiki/Steven_Lev ... n_on_Crime is Levitt's famous work, in which he argues that:

(Levitt) shows that the legalization of abortion in the US was followed approximately eighteen years later by a reduction in crime, then argues that unwanted children commit more crime than wanted children and that the legalization of abortion resulted in fewer unwanted children, and thus a reduction in crime as these children reached the age at which many criminals begin committing crimes.

I think we can agree that Levitt is no more a domain expert in crime than Berri is in basketball, no?

As said, you'd be hard pressed to find an economist who is a domain expert in something they have applied economic and statistical analysis to. is a great talk on the effects of various programs to reduce AIDS, a GREAT watch BTW. I highly recommend it.

Anyway, if we exclude people without specific domain knowledge from commenting on a topic, we'll be left with what?

Predictions. WP48 isn't good at predicting future outcome of games,

Really? That's been shown? I'd love to see that demonstrated. Can you point to an example or article or something similar?

WoW had an article the other day that said prediction doesn't matter. Nuff said.

That's not true. They said you shouldn't be judged on what you DIDN'T know. Take as an example of the point Orlando signing Grant Hill in 2000. Should Orlando's Front Office be judged by the outcome (very, very, very little for Orlando) or by what they knew at the time (Hill was one of the best players in the NBA)?