SCHOENE

[freemoney]

Hi, I am a newcomer to metrics. How exactly accurate are the SCHOENE projections (for both players and teams)?
Have they fared well in the past?

[Kevin Pelton]

Here's what I wrote in this year's book as part of the SCHOENE explanation:

How has SCHOENE performed?
Honestly, not great. Of the six pure statistical projection systems tracked last year on the APBRmetrics message board, SCHOENE had the largest mean error, pegging teams wrong by an average of 5.0 wins over the 66-game schedule. However, when I studied possible adjustments over the summer, I found nothing that would have improved SCHOENE's results over multiple previous years. So the system remains unchanged from last season, which is essentially the third incarnation of SCHOENE. (The first was used only in 2008-09. The second, rolled out for the first edition of Pro Basketball Prospectus in 2009-10, began incorporating multiple years of past player performance.)

Historically, SCHOENE has proven more effective at pegging the direction teams are heading than their specific win total. So a different measure--which system was closest to each team's final record--showed SCHOENE performing as effectively as any of the other systems. SCHOENE was closest to the pin on six teams; only a set of projections using regularized adjusted plus-minus as tracked by poster EvanZ did better, with seven.

The moral of the story is to temper the most extreme projections. When SCHOENE projects that the Minnesota Timberwolves will be an elite team this year, the appropriate conclusion is that the Timberwolves are closer to contending than conventional wisdom would indicate, not that they are as good as anyone in the Western Conference.

What are SCHOENE's weaknesses?
Historically, there are a few types of teams with which SCHOENE has struggled. The projection system tends to be overly pessimistic about veteran teams, in no small part because players seem to be aging better now than ever before. Limiting the pool of comparables to player seasons since 1989-90 helped this issue, but did not completely alleviate it. SCHOENE also tends to favor offensive-minded teams over those with good defenses because of the historical trend that defense regresses to the mean more than offense.

I haven't looked specifically at player performance since after the 2008-09 season. Finding a good benchmark is a bit of a challenge there. At the time, SCHOENE outperformed Basketball-Reference.com's Simple Projection System. I've never had complete stats to compare to John Hollinger's projections. There's also a question of what we're trying to project--fantasy stats? player value?--since those are slightly different issues.

[DSMok1]

As Kevin's post indicates, a player comparable-based projection system is a little unique. Most standard projection systems would take a "look at multiple years, apply standard aging curve, regress to mean/prior = projection". That basic projection system is easier to get "right" on average, but also tells you less about the player. A player comp approach gives more interesting information, handling unique situations as unique, but doesn't do as well nailing the "easy to get right" players.

Most of that previous paragraph is based on how PECOTA has fared vs. other projection systems in baseball, which I have watched over a number of years now.