Forecasts and Brier
Market commentary is unaccountable, and became more so the moment a model would produce a confident directional view on anything you asked it. The problem is not that views are wrong. It is that being wrong costs nothing and leaves no trace, so a forecaster worth reading and one who is merely fluent are indistinguishable from the outside, and from the inside too.
Registering a call
Three things are required before the fact, and none of them can be added afterwards.
| Field | Why it is mandatory |
|---|---|
probability |
A direction with no number cannot be scored. 0.5 is an admission, not an answer. |
horizon |
A call with no deadline is never wrong, only early. |
resolutionCriterion |
Settled by the string, not by argument afterwards. CSPX close > 742.00 leaves nothing
to interpret.
|
Settlement
Criteria that parse into a comparison resolve themselves: a cron sweeps every book's due calls, evaluates the criterion against the stored price series and scores the result. Nothing waits on you remembering that you made a prediction three weeks ago.
The sweep is the one deliberate exception to tenant scoping. It runs on a tenant-less index and is an
internalMutation for exactly that reason, unreachable from any client, with a test asserting
each book is scored against its own prices.
The score
Brier is the mean squared error of the probabilities you gave:
brier = mean((probability - outcome)²) // outcome is 1 or 0| Value | Reading |
|---|---|
| 0.00 | Perfect, and if you see it, check the criteria. |
| 0.25 | What a coin flip earns. The bar, not the goal. |
| > 0.25 | Worse than saying 50% to everything. |
Reading the reliability diagram
A mean Brier alone hides the direction you are wrong in. The diagram buckets calls by what you said and shows what actually happened in each bucket, as two overlaid bars. The gap between them is the entire point.
- Observed below said, at the top end. Overconfidence. The usual failure: your 90% calls land 70% of the time.
- Observed above said, at the bottom end. Underconfidence, which costs less but is still a miscalibration.
- Bars level across every bucket. Calibrated. Your 70% means 70%.
Calibration and discrimination are different things. A forecaster who says 50% to everything is perfectly calibrated and completely useless, which is why the bucket counts are on the page next to the rates.