Contents
125k
Of users across Google
1230m2
Of square feet analyzed
43
Of buildings across Google
How do you tell whether an AI prediction is credible?
Run three tests. Does the claim name a date and a number you could check, does the forecaster publish their misses as well as their hits, and what do they lose if the claim turns out wrong. Most industry commentary fails the first test, which makes the other two unnecessary.
How are forecasts scored?
Forecasting competitions score each prediction with a number. A perfect forecast scores zero, pure guessing scores 0.5, and being reliably wrong scores 2.0. The useful part is that a score only works on claims specific enough to grade, which rules out most public predictions straight away.
Are expert predictions better than random guesses?
Slightly better, and the detail matters. Philip Tetlock gathered close to 30,000 expert predictions and found that experts "edged out the dart-tossing chimp but their margins of victory were narrow". The popular version of this finding, that experts do no better than chance, goes further than the research does.
What makes a forecaster unreliable?
Tetlock found the worst performers shared one habit. They had one big idea and repeated it, whatever new evidence arrived. That habit is easy to spot in AI commentary, where much of the loudest output is a single 2023 idea with newer examples attached.
Why do the forecaster's incentives matter so much?
Because incentives shape forecasts more reliably than intelligence does. An AI lab predicting rapid growth has a company valuation resting on that expectation, and a consultancy predicting that the value lies in implementation sells implementation. Neither is necessarily wrong, but a reader should discount both.
Did Fluxon really predict that AI labs would sell services?
We told our own team about 18 months before it happened that service companies would capture much of the value, because the building work is the hard part. In 2026 Anthropic announced a services company and OpenAI launched a deployment company with $4 billion behind it. We also had a commercial interest in that outcome, which readers should weigh.
What is wrong with citing only your successful predictions?
Someone making many loud claims will get some right by chance alone, so a single hit says almost nothing about skill. Without the full set of calls, including the wrong ones, a genuinely good record and a lucky one look identical from outside.
How should a company publish a prediction record?
Write the claim down before the outcome, with a date and a number attached, and publish the losses as prominently as the wins. A record put together afterwards from memory is close to worthless, because memory quietly edits out the calls that did not work.
Is "AI will replace junior developers" a real prediction?
Not in the form it usually appears. It names no date, no measurement, and no definition of "replace", so no outcome can prove it wrong. Rewrite it as a specific claim about job postings falling by a stated amount by a stated date and it becomes gradeable. Far fewer people are willing to say that version.
What should I do with a prediction that fails all three tests?
Read it as commentary rather than foresight, which is a perfectly reasonable thing to read. The mistake is not reading it. The mistake is treating it as insight and letting it shape a roadmap, a hiring plan or a budget.
What is Fluxon's current prediction?
That by the end of 2027, most production AI work at the companies we serve will run on open models you can download and run yourself, rather than on paid access to a frontier model. We expect frontier models to be kept for a small share of genuinely hard work. If the paid models pull far ahead again, we will have been wrong.
How often should predictions be revisited?
Whenever evidence arrives that would have changed the original call, which is a different trigger from a date in the calendar. The best forecasters update when new information appears. The worst treat sticking to their earlier position as a virtue.
