APBRmetrics

What reasoned belief should one have about the practical value of analytics to an NBA franchise? Crow recently introduced the topic of a notional analytics award, and discussion ensued about the best criteria for assessing performance. Another recent post cane from Houston, seeking applicants for analysis positions n and n+1. There is an implicit, common assumption to these (and indeed most threads): that applied analytics confers significant advantage to participating franchises.

But what evidence is there that this is true? As a historical matter, has there ever been a clear "Moneyball Moment" in the NBA? Then, even if so, what reason is there to believe that opportunities remain for "significant" competitive advantage to accrue to the relatively clever?

The apparently obvious example of a Moneyball Moment is Danny Ainge's fleecing of Kevin McHale in 2007, getting Kevin Garnett and giving up essentially nothing in return. This was a major triumph of analytics, APM in particular. Case closed. Or maybe not.

I remain open to the interpretation that this deal was a particularly effective bit of Zarrenesque sock puppetry, but until the definitive history is written confirming the point, it isn't clear to me at all that Ainge acted because he thought (make that knew) he was acquiring a highly undervalued defensive asset on the cheap. And the contrary view, to my mind, is bolstered by subsequent, rather unenlightened personnel moves. And if Ainge just lucked into a really good thing, based on conventional reasoning, then the deal doesn't fall under the Moneyball category.

Is this the exception that proves the rule, or are there other one-sided trades in the naughts that illustrate the benefits that analytics conferred to the early adapters?

To take a more comprehensive approach however (not focusing on one-offs that might admit differing interpretations) it seems to me that a focus on the Dallas Mavericks is required. If there has ever been an advantage gained from adopting analytics in the NBA, it must be seen in this franchise, not only because it was the earliest adapter (I think) hence the record is long but also because the regime was imposed from the top, hence integrated as best as one could ever expect.

And what is the Maverick record? On the one hand there is a very real accomplishment, not only of competitive excellence but consistency as well. Theirs is one of but a few franchises in NBA history with a decade plus run of 50+ wins. On the other, Mark Cuban happened to buy a franchise with two future HOFers on it, so account must be taken of this initial endowment in explaining its success, well, of Dirk Nowitzki at least.

Leaving the late-blooming Steve Nash out of it, the effect of having Dirk over the 2002 to 2011 seasons (taken to conform to Jerry's 10 year RAPM ratings) was to bequeath the Mavs with a player playing 75% of available minutes and 4.67 points per 100 possessions (a figure that is biased low). Over the same run, the Mavs averaged about 57 wins, equivalent to an approximate edge of 6 points per 100 possessions.

So, the issue then is the degree to which analytics can explain the other 1.33 points per 100 possessions that the Mavs achieved. That is an accomplishment (especially taking place, consistently, over a decade) but one that is diminished by the fact that the Mavs were also big spenders. Over this time frame, Mark Cuban, net of Dirk's salary, spent 14% more than the average NBA franchise (Mavs included).

Is it an analytic triumph to spend 1.14 as much as the average NBA team, only having to purchase 85% of the minutes of the average team, and still be only 1.33 points better than the average NBA team? Netting out the contribution of cold hard cash, there could well be a positive residual, but if so, it must be quite small.

And then looking forward, the general outlook seems far more dismal. First because included in the Mavs history are all the gains from low-hanging fruit never to be picked again. And second, because the Mavs are an institutional anomaly, with the putative benefits from analytics facing the minimum of resistance from those obliged to act upon the information.

Maybe I am thinking about this incorrectly. Maybe there are new frontiers, currently unimagined, just waiting to be revealed and exploited by another Rocket with experience in SQL. But maybe not.

Maybe someone could estimate the average championships expected for superstars of different levels of performance. (Maybe that has already been done somewhere?) If you had that baseline, maybe you could try to estimate how much an organization added to that star talent with coaching and gm work. Of course the star performance and the organization's results are ultimately a joint product but maybe this approach would be some use even if it is not pure and powerful at separating impacts.

1 title in Boston, 1 in Dallas, none in Cleveland or Houston and lots of other places with statistical analytic support. How much credit to give it in Miami this time, last time? And in LA? When should San Antonio start to get counted as a team with "statistical analytic support", around 2009 or before? What did they really have before 2007, when they were actually winning titles? I know they had a hard-working GM, a highly regarded coach, a strong defensive system / culture and future GMs in lower and middle management but did they really have a significant edge on other teams specifically in statistical analytic support? Where did / does that show up distinctly (if one does not want to use actual to expected win efficiency)?

The case for significant impact of statistical analytic support on winning titles is still not that strong or clear-cut. Maybe another 5-10 years of results will make it a bit stronger, a lot stronger or not. Maybe it will not really enhance clarity much.

Why was picking up Garnett especially great for APM? Did/do the Celtics prefer APM to other metrics? Otherwise, Garnett scored very well in most metrics before joining the Celtics. Wins Produced thought he was great, and PER and Win Shares had him as at least very good (while not rated as highly as in WP, he still led the league in PER, total Win Shares, and WS48 a couple years before joining Boston). Perhaps all these other measures underrated him because of defensive considerations, but he was still a great pick-up at 'face value'.

Given his previous MVP award, multiple All-Star games, and multiple All-NBA teams, it also seems fair to say that most people thought he was pretty good outside of using advanced stats.

schtevie wrote:What reasoned belief should one have about the practical value of analytics to an NBA franchise? [...] that applied analytics confers significant advantage to participating franchises. But what evidence is there that this is true?

IMHO the most important aspect of such departments in the front office is just a fresh perspective from people who can constantly question the status quo. Is it the most optimal way to do this? What is the better way to do it? In any other capacity such approach would very easily lead to being fired.

schtevie wrote:As a historical matter, has there ever been a clear "Moneyball Moment" in the NBA?

Unfortunately we still have to wait for the book ;-)

schtevie wrote:Is this the exception that proves the rule, or are there other one-sided trades in the naughts that illustrate the benefits that analytics conferred to the early adapters?

IMHO the most glaring examples are with the draft picks. How teams handle them, how teams trade them, what teams do with them outside of the lottery etc.

schtevie wrote:And what is the Maverick record? On the one hand there is a very real accomplishment, not only of competitive excellence but consistency as well. Theirs is one of but a few franchises in NBA history with a decade plus run of 50+ wins.

IMHO success on a team level is a terrible way to judge effectiveness of one particular department in the front office because it assumes they have control over all the decisions and I seriously doubt that's a case. That's probably the biggest problem with the whole question: internally Mavs can easily judge how analytics helped them or not because they know what was their input but that's not true for the outsiders. If they sign someone terrible according to measure X it can mean they don't use it or... they overruled it based on other factors.

schtevie wrote:Maybe I am thinking about this incorrectly. Maybe there are new frontiers, currently unimagined, just waiting to be revealed and exploited by another Rocket with experience in SQL. But maybe not.

I think there clearly are... like for example optical tracking.

Over the last 22 seasons, about 8.1 guys who played 30+ minutes per game were rated at .200+ winshares per 48 per season. 28 won titles in those 22 years or about 1.3 per season. Thus it appears that a .200+ winshares per 48 per season coincides with winning a title about once in every 6 cases on average.

Dirk has had 8 .200+ winshares per 48 seasons so far. So his ratio of titles per such seasons is a bit below the league average.

Kobe has had 6 .200+ winshares per 48 seasons so far and 3 near misses. 5 titles. In only 2 of them was he actually +.200+ winshares per 48. Shaq over .200 in 3 of 4. Duncan 4 for 4. Ginobili 2 for 3.

Could find the odds of winning a title when you have 2+ stars over .200+, but I'll settle for assuming it is higher for now. It has happened 8-9 times in the period, including for 3 of the last 5 titles.

A few further words on Maverick performance accounting:

First, I want to elaborate on and unmuddle some of my remarks on notional counterfactual salary expenditures. The issue is how much should the Mavs, having Dirk Nowitzki in the fold, have had to pay to achieve the level of success it did: 57 games won per year in the 10 year period in question.

According to Jeremias, between 2002 and 2011, Dirk when playing provided +5.6 points per 100 possessions, and he played 75% of the minutes on average. This implies (if I am getting the math right) that with 3.25 full-time players with RAPM of 0 and one full-time player equivalent providing +1.8, this would be a full roster, adding up to a +6.0 team, that in expectation would deliver approximately 57 wins.

Daniel recently provided a formula, relating RAPM and share of minutes played to salary: (RAPM + 3.25) * (share of floor time) * ($3.8 million) = salary.

Using this to run the numbers for non-Dirk inputs, expected non-Dirk salary totals should have been $59.3 million. If you look at 2010 numbers, when Dirk's salary reached its peak, non-Dirk salaries totalled $68.6 million, suggesting an overpayment of $9.3 million. And for other years the gap would be quite a bit higher still.

Now, I don't know if Daniel's formula is exactly fair for the counterfactual at hand. What we need to know is the price for talent that is available on the market. As such, the RAPM/salary relationship should properly take into account the realities of the CBA, which puts a ceiling on payment to rookies and stars. I don't know if taking these effects into account would fully account for this "overpayment", but it is an important caveat.

And then there is another general factor that must be explicitly considered in accounting for the long-term performance of the Mavericks, and that is the cummulative disadvantage that successful teams face by being chronically allotted low draft picks. I am not exactly sure how to best handle this aspect, but my sense is that it isn't that important, in two senses.

First, their general strategy seemed to be to trade away draft picks, recognizing that the bottom twenty slots aren't going to produce much value. (Implicitly, I suppose that this means that they didn't think their special analytic sauce could be applied to the draft. Perhaps someone here has some expertise on the Mavs philosophy on drafting). In which case, the appropriate measure of their success is how well they did with their money on the market.

There was however a singular exception to their lack of apparent interest in the draft, and that was getting Josh Howard at #29 in 2003. This was a steal, and the interesting thing is what this implies for the larger analysis. On the one hand it should be chalked up to the analytic prowess of the Mavs organization. If so, however, this has direct implications on how well the Mavs spent their money over the period.

Josh Howard wasn't with the Mavs for the entire ten year period of this evaluation, but we have been looking at averages throughout as this is what we care about, so we can create an average Josh Howard, whose effect was evenly distributed over the entire period. During his stint with the Mavs, he had an average RAPM of +2.7, and then if you average his playing time across ten years, it covers 34.5% of available minutes.

Anyway, long story short (and I hope I got the math right) if you factor in this averaged Josh Howard, in order for the Mavs to get to +6 as a team (again, given the presence of Dirk Nowitzki), instead of needing one full year of a +1.8 player, you only need 0.655 of a year of a +1.3 player.

And this then implies that the Mavs should have spent only $51.5 million for non-Dirk, non-Josh Howard salaries. Of course you also have to add back what Josh Howard was actually paid on average, but this was peanuts ($2.8 million worth of peanuts) as most of the years he was constrained by his rookie contract.

To try and put this in a single year context, once one takes into account the benefits of an "averaged Josh Howard" and the inherited gift of an actual Dirk Nowitzki, for 2010, Daniel's formula implies that the Mavs "overpaid" for talent by $14.3 million.

Stipulating that Daniel's salary formula is approximately correct, what is the argument that analytics yielded the Dallas Mavericks a non-negative return?

And if it didn't pay off for the Mavs - over a ten year period - who began when the low branches were laden with the largest, ripest fruit and for whom there was the maximum institutional support for thorough implementation of the program, what rational belief is there that it will pay off for others?

My formula for dollars per performance was based on last year's salaries and varies based on the overall payroll of the league (linearly).

Salaries:

Yes and no. I believe when it comes to Moneyball-ing the NBA, the key thing is going to be the theory and philosophy behind the moves, not the number crunching.

I believe a growing number of teams are moving in an "trade asset accumulation" direction. Houston is the most obvious example right now as a team who's not worrying about the fit of their players as much as just getting as many high value pieces as possible. When players have a high trade value it's for a reason. Talent, youth, contract, etc. Multiple value pieces can be turned into a bigger one, can be used to move up in drafts, etc. Because every asset a team has can be measured by trade value and can be moved for other assets that fit better, the teams who have the most should be in the best position. For these teams small assets matter a lot because they can be added to bigger deals, or just increase the trade value on their own. Cleveland has been the masters lately of using capspace to "buy" these small assets and extra picks, which is a much more efficient use of capspace trade value wise than spending it on FAs, which are typically overpaid, especially for a destination like Cleveland or Toronto. Oklahoma City of course has also been very progressing trying to use capspace efficiently - the Ibaka, Aldrich, Thabo acquisitions were from being ahead of the curve in this way, and I don't believe they pull off the Perkins deal without having a future LAC 1st in their back pocket to throw in with Green. What OKC has done remarkably well is in general, maximized the efficiency of their assets. Their top 5 picks turned into stars, their capspace got them high trade value young players. They were an investor getting a killer average return. This is what it's all about. You have assets (picks, capspace, players, etc.) - How efficiently can they be used?

In regards to trade value, to use an example I do a blog post once a year where I rank the total trade value of all a team's assets (including future draft picks). In February 2011 I ranked the Clippers 5th, which was a strikingly high ranking for a team in the middle of a 32-50 season who hadn't been good forever. But their trade assets at the time were insane. They had Griffin during his crazy rookie season, Gordon during his breakout year, Jordan, Aminu when he had some value, Bledsoe, 2011 Clips 1st (a few days later traded to CLE), and MIN's 2012 1st which looked like a top 5 pick at that point. Of course what happened by the next December is that all these assets came to fruition for the Clips by allowing them to get Paul. And this was despite a somewhat inefficient use of their assets, first by stupidly blowing away a 1st just to swap Baron Davis for Mo Williams, then by (IMO) overpaying compared to the next offer to get Paul, I think NOH should've been happy with just Gordon and Aminu. So this is an example where the Clippers obvious top 5-6 trade value in 2011 reflected they were in a terrific place as a franchise, despite their low record. I believe any trade value that ends up that high for a team is going to end up with them successful, barring a disaster like Portland's career ending injury flu.

The thing is there's still a strong line in the sand between teams who understand this and teams who don't. I mentioned Cleveland, but take Toronto, who is a good comparison because they were in a similar position in the summer of 2010 (their superstar left and they had to rebuild). The Raptors under Bryan Colangelo as a whole have been a trade value accumulation apocalypse. To give an example Toronto traded TJ Ford, Rasho and a 1st round pick (that Indy used on Hibbert) for Jermaine O'Neal. This was a terrible trade value move because O'Neal, even disregarding his injury history, was so overpaid that he didn't have much value on the market, and he was to be a UFA after 2 years, while being old. It was essentially emptying the bank account to buy a stock before it's about to fall off a cliff. Ford, despite his injury, was still a youngish PG, and a 1st round pick speaks for itself. It also killed the team's capspace. Half a season they traded another 1st just to get rid of Jermaine O'Neal, so they could make a run at Odom, Hedo, Ariza in the offseason, none of which were likely to have eventual trade value. Colangelo also killed the team's capspace repeatedly with other moves like spending all the team's capspace on veterans in Parker/Garbojosa/Rasho in 06, Kapono the year after, Turkoglu in 2009. He does not understand capspace is a trade value building asset, whereas Cleveland's entire gameplan the last 2 years has been to use capspace as a trade asset. The summer of 2010 was particularly stupid for Toronto though. For one thing Bosh was allowed to leave in UFA for virtually no value (a couple 1sts in a S&T) despite everyone and their mother knowing he wanted out. We actually turned down Michael Beasley (coming off his 2nd season, still had value/upside) in the S&T for Bosh, again another strike for "doesn't understand trade value". The team then signed Linas Kleiza and Amir Johnson to a combined 50 million over the next 4-5 years, despite both being veterans on a team that didn't need to win for years. While Cleveland has spent the last 2 years trading capspace for picks, Toronto spent it on veterans and overpaid MLE guys. This offseason was more of the same. Lowry is a good player but a bad team should be wary of giving up a lotto 1st for him, because Lowry is a UFA in 2 years and could very well leave for nothing. Landry Fields was yet another disregard of capspace as a trade asset and he, Kleiza and Amir will be making 15 mil+ a year combined for virtually dispensible play, all locked up for multiple seasons. The team now Jerryd Bayless was renounced, giving the team no value in return for a player that likely could've fetched some at the TD. None of this indicates a team who realizes the concept of stockpiling trade assets - like at all.

This isn't quite Billy Beane's A's, but it's an indication of a strategy one team is taking while half the NBA or more, are sort of dinosaur-ing it. I have a feeling that if teams started to make decisions based on the idea of "maximizing the efficiency of every asset", many moves would be different

Good post, Dr. Positivity. You have a blog?

In what trade that has happened did Houston effectively use the "trade asset accumulation" strategy? The only one I can see any case for is for Marcus Camby and that sure was significant for re-tooling the Rockets into a playoff participant and contender, right?


How many assets have the Rockets waived in the recent summers because they had too many "assets"? I count at least 13 in 6 years, not counting routine scrub adds & deletions.

DSMok1 wrote:Good post, Dr. Positivity. You have a blog?

Is it asubstituteforwar.com?

http://asubstituteforwar.com/

It's ok, I'm a philosophy class guy and not a stats one and kind of lazy, so it might not have the typical data backing you guys like

On the topic of market inefficiency, I also wanted to share what I have become convinced is one of the biggest market inefficiencies out there

IMO talent is weighted all wrong in the NBA. I'm obsessed with the NBA Draft, but what I've realized is that if we were to split up how much teams care about physical tools, skill level, and feel for the game, I believe they draft with a split of something like 70% physical tools, 25% skill, 5% feel for the game. To give some examples in this year's draft

Andre Drummond - Amazing physical tools, 0 skill or feel for the game. 9th overall, but consensus top 3 talent/upside
Thomas Robinson - Great physical tools, average/weak skill and feel for the game - 5th. Andrew Nicholson - weak physical tools, elite skill/feel for the game - 19th. This is the most obvious example of how physical tools is weighted vs skill/feel. It's almost a perfect comparison. If one sees physical tools as 2x skill/feel combined for talent, Robinson 5 Nicholson 19 makes sense. If one see skill/feel combined as 2x Robinson, then Nicholson 5 Robinson 19 is what makes sense. I had Nicholson 5th Robinson 21st on my personal big board this year.
Austin Rivers - weak feel for the game, Jeremy Lamb - elite feel for the game. Rivers 10th Lamb 12th. Rivers over Lamb was absolutely ridiculous to me. Lamb is much bigger, more explosive, a much better shooter and has 10x the feel for the game, wtf does Rivers have over him?
T Zeller, Fournier, Sullinger, Crowder, Draymond Green, Doron Lamb = great feel for the game, drafted 17, 20, 21, 34, 35, 42. Tony Wroten - elite physical tools but no skill or feel or the game, still goes 25 above Crowder/Green/Lamb
Scott Machado = truly special feel for the game, undrafted
Valanciunas hyped up as "#2 if he had been in 2012" despite having horrible feel for the game

In previous drafts, John Wall hyped up as superstar #1, Irving seen as low upside/non all-star. Wall has average/weak feel for the game and skill but elite physical tools. Irving incredible skill and feel for the game, lesser physical tools.
Ekpe Udoh picked ahead of Greg Monroe, Al-Farouq Aminu picked ahead of Hayward and George, Thabeet picked ahead of Harden. Udoh/Aminu/Thabeet all have serious feel for the game/skill problems, Monroe/Harden/Hayward/George skilled with feel but not as flashy physically.
Ty Lawson with elite feel for the game and good skill goes 18 behind non feel physical talent first players like Jordan Hill, Terrence Williams, Earl Clark, etc. and of course, Jonny Flynn

I also know from someone who reads too many draft profiles and write-ups, that "feel for the game" never EVER gets mentioned as a strength or weakness for a project. I don't think I've read Chad Ford write the words feel for the game, ever. And this is a process where every prospect gets hyped up for one thing or another.

Most of the draft busts fit a particular profile. They are physically gifted (athletic, long, etc.) but have weak skill or feel. So they end up journeyman bench players. I believe this is because physical tools are weighted at 70% for teams. If physical tools were really 70% of what mattered talent wise, a freak athlete could be a starter based on that alone. But we know that doesn't happen. Teams continue to draft players believing that they are unsculpted pieces of rock that all have an equal chance of adding skills. I believe this is fundamentally wrong. One is that many skills are apparent early. Lots of post players and shooters or on ball handlers stood out at it as freshman in the NCAA. Secondly is that feel for the game is essential to being a skilled/finesse player and that's something that players have or they don't have, just as much as they have athleticism or they don't, IMO. Let's say feel for the game was half of what makes a "skill player", and development can only take you halfway towards standout in a skill area, the rest has to be innate/talent based. That would mean development can only account for 25% of skill, while 75% comes from talent and a "has it or not" place. This is why Drummond isn't going to turn into Amare. If the numbers I just threw out were correct, he has a chance to get the "development 25%" but the feel for the game 50% is basically off the table, as well as the 25% of skill that comes from talent. This is of course disregarding the fact that Drummond is reported to be not the type of worker to add all those development points. Most likely scenario is he ends up as bad as Tyrus Thomas offensively

I believe physical tools, skill and feel for the game is close to a 33/33/33 split. I've ran a lot of grades on players giving my subjective ranking of them in the area, and it basically grades out for every player I've done (I know that leaves open a "the results created your scores" door, but w/e). I've also done evaluations on how strongly teams do in each of the areas and the teams stack up essentially exactly as they should and what most are predicting, save a few outliers (eg. Orlando ranks as a 35 W+ to me because I have them as actually a top 3-4 skill team in the league with pretty good feel for the game, so even with the worst physical score in the league by far, the fact that it only makes up 33% of my evaluation vs the 66% they do good in, lets them be respectable. Spurs also blow away the field in the West (60 Ws vs 53-55) and MIN's score is very disappointing, around 35 Ws. POR ranks as 40 W caliber team)

Even disregarding all this, it seems inherently obvious that physical tools have long been overrated in the NBA and the skill and mental portion of the game has been underrated. It's a lot simpler task for people to connect how physically gifted someone is = talent than say, ability to add skills or how natural/a strong feel a player has. But the reality is that sports are just about the only area in the world where the "feel" isn't considered the biggest part of talent. When it comes to musicians, writers, painters, speakers, etc., just about the only thing that matters in regards to talent is "feel for the game". I would venture a lot of the best authors in the world from a young age sat down and great stories just came out of them. Taylor Swift could write #1 charting pop songs when she was like 13 and probably an idiot. Charisma and social skills are in a way, feel for the game. The NHL among major sports actually is the one that recognizes that some players just have a vision and feel for where everyone is spatially and that this is key to playing on both ends (there's still some market inefficiency there I suppose though, Grantland had a good article a while back showing how small players were probably underrated vs the big bruisers) It's a real thing that the NBA tends to ignore. That's what I think the market inefficiency is. Move Physical tools from 65-70% to 30-35%, and move Feel from 0-5% to 30-35% and a lot of evaluation changes, mostly in the draft but outside of it too

Crow wrote:In what trade that has happened did Houston effectively use the "trade asset accumulation" strategy? The only one I can see any case for is for Marcus Camby and that sure was significant for re-tooling the Rockets into a playoff participant and contender, right?


How many assets have the Rockets waived in the recent summers because they had too many "assets"? I count at least 13 in 6 years, not counting routine scrub adds & deletions.

Off the top of my head

Dealt Landry for Kevin Martin
Got Jordan Hill, future 1st from NY for Tmac, when NY wanted to dump Jeffries for the Lebron capspace
Got future 1st, Dragic from Suns for Brooks
Got future 1st, Thabeet for Battier
Traded Hill for 1st, Budinger for 1st, Lowry for lotto 1st
They also took fliers on Flynn and Terrence Williams (cost a 1st) that didn't work out

Furthermore this summer alone is indication. They have like 7-8 PFs just because they're on the team to be assets, they got those 3 picks in the teens to try and move them for Dwight Howard to move up in the draft. It's also very clear that their plan leading up to the Paul-Pau void was to build assets for a while and then turn them into a superstar (they tried it twice so far and it hasn't worked)

Matt or Julien? I read ASFW.

It would be interesting to try to work towards a coherent framework for measuring assets. I think we're most of the way there. I have compiled marginal value of draft picks ( http://godismyjudgeok.com/DStats/APBRme ... 31637.html ) as a start...

The key for asset accumulation is actually using them to get something than ordinary teams can't or don't. They haven't.

Dealt Landry for Kevin Martin (nothing special there)
Got Jordan Hill, future 1st from NY for Tmac, when NY wanted to dump Jeffries for the Lebron capspace (what greats things did they get for Hill? Derek Fisher-waived and a conditional first-round pick.)
Got future 1st, Dragic from Suns for Brooks (ok, then what did they get for Dragic? nothing)
Got future 1st, Thabeet for Battier (forget about Thabeet, who did nothing, this is another, nothing special 1 for 1)
Traded Hill for 1st, Budinger for 1st, Lowry for lotto 1st (again completely normal 1 for 1 stuff)
They also took fliers on Flynn and Terrence Williams (cost a 1st) that didn't work out (yeah totally predictable flops)