Extreme Forecasts

The Basics

Data Used:

• Year 2 binary IFP's, answer option 'a'. Total of 73 IFP's, 1221 forecasters, 101,565 forecasts made.

Definitions:

• Extreme Forecast - a forecast of 0 or 1 (on a binary IFP's option 'a')
• Extreme Forecaster - a forecaster who submits at least one extreme forecast
• Active Extreme Forecasters - extreme forecasters who have sumbitted at least 10 forecasters in year 2
• Extreme Only Forecast - the event where a forecaster submits only extreme forecasts for the duration of an IFP

First we'll look at the forecasts made on option 'a' for binary IFP's in year 2. We'll consider extreme forecasts to be a forcast of 0 or 1, since a forecast of 0 on option 'a' is necessarily a forecast of 1 on option 'b'.

In year 2 there were 746 (61% of 1221) forecasters who made at least one forecast of exactly 1 or zero on outcome 'a' of a binary IFP. There were 9,008 extreme forecasts submitted (8.9% of all forecasts), with at least one on every (binary) IFP.

This graph shows the proportion of forecasts made which are extreme, given that the person is and extreme forecaster and made at least 10 forecasts throughout the whole year.

Extreme forecasts across all users:

Looking at the proportion of extreme forcasts out of all forecasts made:

Table 1: Proportion of Extreme Forecasts by Group & Total (All Users)

Table 2: Proportion of Extreme Forecasts by Group & Total (Active Extreme Forecasters)

Who Makes Extreme Forecasts?

Expertise

Is there any relationship between expertise and extreme forecasts? I.e. are forecasters more likely to make an extreme forecast when they believe that they have expertise on the subject?

There does seem to be some relationship between expertise and extreme forecasts. More of the extreme forecasts seem to be made by those at level 5 than at any other expertise level.

Group (Individuals vs. Teams vs. Superforecasters)

Who is more likely to make the extreme forecasts?

So by group, super forecasters are more likely to use an extreme forecast. Individuals and teams use the extreme forecast more sparingly.

Do extreme forecasters make less forecasts?

When are extreme forecasts made?

Extreme forecasts tend to be made closer to the close date of an IFP.

Extreme forecasts are not more (less) likely to be made on a long (short) IFP.

Brier Score

Do forecasters who make extreme forecasts have lower brier scores? At first glance it would appear that the answer is no. Simply using an extreme forecast at least once doesn't seem to improve Brier score.

However, on average forecasters have a significantly lower average Brier score on the IFP's where they made at least one extreme forecast than they did on other IFPs.

Attempting to understand the relationship between Extreme Forecasts & Brier Scores

Only Extreme Forecasts

What happens to a forecasters Brier score if we consider IFPs where a forecaster made ONLY an extreme forecast? There were 2,663 instances (combination of user by IFP) from a total of 45,249 (5.8%) where a user made only an extreme forecast. There was at least one forecaster who used only extreme forecasts on every IFP, and this type of forecast was made at least once by 569 of 746 Extreme forecasters (77%), 47% of all forecasters.

So do people who are “more sure” earn lower Brier scores? Yes. The mean Brier score for the combinaion of IFP / user when only an extreme forecast was made was 0.33 vs. 0.38 for all others. This difference is even more pronounced when we look only at users who made more than one forecast on an IFP, then the extreme only forecasts mean brier score drops to 0.19 and the non extreme drops only to 0.32.

This difference holds across each of the groups, but the difference is strikinging in superforecasters who seem to only make extreme-only forecasts when they are very sure of the outcome.