Investigating where player metrics fail

[J.E.]

I've developed a tiny tool that should give us, for each metric, those players the metric has (had) a bad read on. If we see certain patterns in top over-/underrated list of players from year to year, that should give us hints on how to improve our metrics

What I do is this: Take player ratings from season X-1, then run through all 5on5 matchups in season X and compare forecasted offensive efficiency with actual offensive efficiency. Some up the difference between the two for each player. The perfect metric should yield an error sum of 0 for each player. Players that the metric underrated should get a high positive (actual-forecast) etc. The players at the top and bottom of the list (most underrated and most overrated) are those that are most responsible for the forecasting/retrodiction error values not being lower

Here are the results for 2012 (forecasted with '11 RiRAPM)

Code: Select all

Overperformed on offense compared to forecast
Patrick Mills 
Jordan Hamilton 
James Singleton 
James Anderson 
Jordan Williams 
Matt Bonner 
Eric Gordon 
Manu Ginobili 
Tiago Splitter 
Nate Robinson 
Jeff Teague 
Jamaal Magloire 
Travis Outlaw 
Al Horford 
Josh Smith 

Code: Select all

Underperformed on offensive compared to forecast
E'Twaun Moore 
Bill Walker 
Terrel Harris 
JaJuan Johnson 
Brian Cardinal 
Sam Young 
Yi Jianlian 
Dominique Jones 
DeSagana Diop 
Jeremy Pargo 

Code: Select all

Overperformed on defense compared to forecast
Ricky Rubio 
Sasha Pavlovic 
Jeremy Lin 
Landry Fields 
Bill Walker 
Ryan Hollins 
Steve Novak 
Jimmy Butler 
Jamaal Magloire 
Larry Sanders 
Ben Uzoh 
John Lucas 
Lavoy Allen 
E'Twaun Moore 
Tony Battie 
Josh Harrellson 

Code: Select all

Underperformed on defense compared to forecast
Jordan Farmar 
Jonny Flynn 
Shawne Williams 
Mehmet Okur 
MarShon Brooks 
Shelden Williams 
Cory Joseph 
Luke Babbitt 
Anthony Morrow 
Deron Williams 
Joel Przybilla 

and '13 (forecasted with '12)

Code: Select all

Overperformed on offense
Tyson Chandler 
Raymond Felton 
Ronnie Brewer 
Kobe Bryant 
Michael Kidd-Gilchrist 
Carmelo Anthony 
Kyrie Irving 
Tristan Thompson 
Dion Waiters 
Pau Gasol 

Code: Select all

Underperformed on offense
Kevin Garnett 
Tyreke Evans 
David West 
Jason Thompson 
Jeff Teague 
Roy Hibbert 
A.J. Price 
Louis Williams 
Dorell Wright 
Paul George 
Trevor Ariza 
Trevor Booker 

This type of analysis really has to be done over a lot of years to possibly draw conclusions, so I'll try to compile a list with more years soon.

When there's groups of players at the top or bottom it's probably a hint that the coach is doing either a great or horrible job

[Crow]

I am interested in this and something I wrote recently is somewhat similar in thinking in a roundabout way. If you do more on this topic, let us know.

Overperformed on offense compared to forecast list appears to have mostly jump shooting perimeter players.

Most on Overperformed on defense compared to forecast
had fairly good steals rate or blocks rate.

Multivariate regression would produce more and better findings than a quick scan.