Page 12 of 13
Re: 2022-23 team win projection contest
Posted: Tue Mar 28, 2023 8:01 pm
by Crow
Alright, thanks for the replies.
(I saw a flattening method article that used medians, a choice for them but not relevant here.)
So minutes from last season matter, a proxy for projection but not projection per se.
Entries and exits from top 9 on minutes or matter but are not projected. Share of current minutes can change / matter but are not projected.
Re: 2022-23 team win projection contest
Posted: Tue Mar 28, 2023 8:15 pm
by v-zero
Yeah. I do think it could easily be improved by trying to better guess at changes in player minutes from one season to the next, but the flattening of the averages definitely helps in this regard.
For what it's worth my daily predictions do use a minutes allocation algorithm.
Re: 2022-23 team win projection contest
Posted: Wed Mar 29, 2023 11:08 pm
by Crow
5 of 6 ahead of me by average error fell back from Mar. 24 to Mar. 27, while I improved modestly.
For recent 8 year period, the top averaged about 5.5 on average error. Bottom about 7.5.
Re: 2022-23 team win projection contest
Posted: Thu Mar 30, 2023 2:29 am
by DarkStar48
Mike G wrote: ↑Tue Mar 28, 2023 6:54 pm
DarkStar48 wrote: ↑Sun Mar 26, 2023 9:45 pm
... it’s actually a consistent finding in the forecasting literature that the simple arithmetic mean (i.e., with equal weighting) of different forecasts often provides highly accurate overall predictions...
Of course, not all forecasts are improved by sheer number of forecasters. The ESPN experts I think numbered 15-20, but they aren't among the leaders.
avgA is the error produced by averaging the predictions of our 6: emin, ncs, dtka, trzu, vzro, and Crow.
Obviously, the avgA “entry” composed of APBR.org submissions is different from a true ensemble average.
The ensemble is incredibly useful in reasonably extrapolating the highly uncertain outcome of a final team record.
Moreover, the robustness of ensemble projection is driven by:
a) Each component has equal weight, so a prediction isn’t given precedence over the others.
b) The combination is based on a large number of individual forecasts with each consequently having a relatively small weight, such that no single one overwhelms the overall mean.
(See the full “Average” ensemble from the 2022-23 season. Link:
https://i.imgur.com/XDACgyj.png)
Re: 2022-23 team win projection contest
Posted: Thu Mar 30, 2023 1:31 pm
by Mike G
That link show's last year's results, and the Average ranked 7th of 30. I'd guess that might be typical.
This year, avg of the whole field of entries would rank 3rd right now in MAE -- dropping one spot, just ahead of Pelton. And 4th in RMSE. So yeah, it seems kind of immune to overall 'bad judgement'.
In the exercise I posted 2 days ago (Mar.28), only 5 of the other entries improved the avgA error by being (individually) included; 8 made the error worse. Not counting dummy entries.
Can you explain what makes a "true ensemble avg"?
Re: 2022-23 team win projection contest
Posted: Mon Apr 03, 2023 10:23 am
by Mike G
Just 3-4 games left per team.
Code: Select all
. avg err rmse r^2 avg err rmse r^2
dtka 4.90 5.90 .67 EBPI 5.94 7.30 .51
avgA 5.24 6.21 .63 MPra 5.96 7.20 .63
vzro 5.41 6.83 .55 538R 6.22 7.78 .49
KPel 5.53 6.50 .58 TmRk 6.53 7.69 .51
LEBR 5.53 6.23 .62 vegas 6.54 7.75 .52
DRKO 5.62 6.59 .58 nuFi 6.65 8.01 .49
ncs. 5.75 6.81 .55 22Re 6.67 8.02 .36
emin 5.80 7.30 .53 EExp 6.68 8.07 .50
Crow 5.80 6.58 .62 AnBa 6.71 7.94 .53
BIPM 5.83 7.19 .67 4141 7.32 9.98
trzu 5.89 6.76 .61 538E 7.59 9.05 .31
UPDATE Apr.6
Code: Select all
. avg err rmse r^2 avg err rmse r^2
dtka 4.81 5.79 .69 MPra 5.84 7.04 .64
avgA 5.07 6.10 .65 EBPI 5.85 7.23 .52
vzro 5.36 6.83 .56 538R 6.17 7.72 .51
DRKO 5.48 6.46 .60 TmRk 6.44 7.60 .52
KPel 5.48 6.52 .59 vegas 6.44 7.66 .53
LEBR 5.49 6.19 .63 EExp 6.57 7.97 .51
emin 5.64 7.19 .55 nuFi 6.60 7.96 .50
Crow 5.67 6.43 .64 22Re 6.65 8.09 .36
ncs. 5.70 6.75 .57 AnBa 6.70 7.87 .54
trzu 5.71 6.60 .63 538E 7.43 8.99 .32
BIPM 5.78 7.12 .67 4141 7.52 10.11
BIPM and EExp with their smallest errors of the season.
Re: 2022-23 team win projection contest
Posted: Mon Apr 03, 2023 11:42 am
by v-zero
Mike G wrote: ↑Thu Mar 30, 2023 1:31 pmCan you explain what makes a "true ensemble avg"?
I guess what Darkstar means is that it isn't an ensemble that includes all participants. That in no way makes it not an ensemble, though, an ensemble is simply an average (equally weighted or otherwise) of predictions/models which have not been created within the exact same framework (i.e. there is at least some variability between them). Ensembles work because they in general reduce both the bias and variance of the predictions when compared to the individual models. However, it is important to note than an ensemble of strong predictors (like you have created) would not benefit from the addition of considerably weaker individual predictions.
Re: 2022-23 team win projection contest
Posted: Tue Apr 04, 2023 7:35 am
by DarkStar48
There is no hard and fast rule in modelling, but I think that a true ensemble combines all reasonable forecasts that are available from different component methods.
The projections need to be non-absurd, i.e.:
-prediction means are not highly irregular, for example, total team wins that are significantly greater than 1230 for an 82-game schedule — slight allowance for asymmetric distributions.
-confidence intervals (when available) do no exceed the possible outcomes, like team wins greater than 82 or less than 0 in a regular NBA season.
The components should include, if possible:
-expectation models (such as, popular polls, betting markets, expert judgments, etc.)
-statistical projections (that is, retrospective, prospective or mixed)
-naive forecasts (e.g., the 4141 “projection”)
Re: 2022-23 team win projection contest
Posted: Tue Apr 04, 2023 12:51 pm
by v-zero
That's ok as your personal definition of an ensemble, but it isn't what ensemble means in a machine learning/statistical framework.
For instance, every day I (well, one of my servers) build lots of prediction models. The majority get discarded during a validation process. What remains makes up an ensemble.
Re: 2022-23 team win projection contest
Posted: Wed Apr 05, 2023 6:29 am
by DarkStar48
@v-zero, I agree.
However, in my experience, it is very easy to make arbitrary decisions on which projections to select and which ones to leave out.
Therefore, as long as a forecast "makes sense," my inclination is to include it (even "bad" ones), so as to make the ensemble in the long-run more robust in response to novel scenarios and data not in the training set.
Essentially, more or less the same logic of gradient boosting.
Still, there can be some benefit in the near-term to weeding out the weaker models to combine the "fittest" ones that survive.
Re: 2022-23 team win projection contest
Posted: Mon Apr 10, 2023 1:28 pm
by Mike G
Final standings!
Code: Select all
. avg err rmse r^2 avg err rmse r^2
dtka 4.67 5.63 .694 BIPM 5.74 7.13 .672
avgA 4.90 5.86 .662 MPra 5.93 6.99 .647
vzro 5.14 6.61 .568 538R 6.10 7.61 .509
LEBR 5.20 5.97 .645 TmRk 6.14 7.40 .537
DRKO 5.26 6.15 .620 vegas 6.15 7.48 .541
Crow 5.33 6.20 .652 EExp 6.33 7.75 .528
ncs. 5.39 6.47 .585 nuFi 6.42 7.72 .516
KPel 5.40 6.34 .587 22Re 6.50 7.83 .370
emin 5.47 6.93 .567 AnBa 6.60 7.79 .546
trzu 5.49 6.42 .637 538E 7.23 8.68 .343
EBPI 5.63 7.05 .531 4141 7.47 9.86
dtka, Crow, and EExp end up with their lowest errors of the season.
Re: 2022-23 team win projection contest
Posted: Mon Apr 10, 2023 5:10 pm
by Crow
Congrats to dtka for the win.
Congrats to V-zero for next best contest entry and all of us in the 2nd best overall Apbrmetrics average.
Thanks to Mike for the scorekeeping.
I finished 6th / 5th / 4th across the 3 measures. Only 2 were better on all 3 and only 2 more better on 2. Not quite what I hoped but pretty decent.
Re: 2022-23 team win projection contest
Posted: Thu Apr 13, 2023 8:03 pm
by Crow
As noted in mid-season, Pelton projected only one 50 plus game winner. The 8 year average was 6 and it turned out to be 6 this season. The long time leader was highly regressed and appeared to benefit from that but in the end finished 8th partly because of that.
Re: 2022-23 team win projection contest
Posted: Thu Apr 13, 2023 10:01 pm
by v-zero
The extent to which Pelton's predictions fell back, admittedly by a fine margin in a competitive pack, does surprise me.
Whilst I did well on the absolute error, I feel it was more a fluke, I wasn't all that close to the top on RMSE, into which I put more weight.
Re: 2022-23 team win projection contest
Posted: Fri Apr 14, 2023 1:03 am
by Mike G
This season, team avg departure from 41 wins was 6.95. Almost everyone went beyond that in their predictions.
Code: Select all
depart.-41 err
KPel 6.0 5.40
vzro 7.1 5.14
LEBR 7.4 5.20
EBPI 7.6 5.63
ncs. 7.6 5.39
DRKO 7.7 5.26
538E 7.7 7.23
dtka 8.3 4.67
538R 8.7 6.10
emin 8.7 5.47
nuFi 8.8 6.42
TmRk 9.1 6.14
Crow 9.3 5.33
trzu 9.3 5.49
vegas 9.4 6.15
EExp 9.5 6.33
AnBa 9.5 6.60
BIPM 10.0 5.74
MPra 10.1 5.93
The positive correlation is .34
Pelton lost his lead when the Pacers, Blazers, Wizards kept losing, and the Dubs, Lakers, Sixers started winning.
