Tech Talk: Graphs as Matrices

Run three tests. Does the claim name a date and a number you could check, does the forecaster publish their misses as well as their hits, and what do they lose if the claim turns out wrong. Most industry commentary fails the first test, which makes the other two unnecessary.

Forecasting competitions score each prediction with a number. A perfect forecast scores zero, pure guessing scores 0.5, and being reliably wrong scores 2.0. The useful part is that a score only works on claims specific enough to grade, which rules out most public predictions straight away.

Slightly better, and the detail matters. Philip Tetlock gathered close to 30,000 expert predictions and found that experts "edged out the dart-tossing chimp but their margins of victory were narrow". The popular version of this finding, that experts do no better than chance, goes further than the research does.

Tetlock found the worst performers shared one habit. They had one big idea and repeated it, whatever new evidence arrived. That habit is easy to spot in AI commentary, where much of the loudest output is a single 2023 idea with newer examples attached.

Because incentives shape forecasts more reliably than intelligence does. An AI lab predicting rapid growth has a company valuation resting on that expectation, and a consultancy predicting that the value lies in implementation sells implementation. Neither is necessarily wrong, but a reader should discount both.

We told our own team about 18 months before it happened that service companies would capture much of the value, because the building work is the hard part. In 2026 Anthropic announced a services company and OpenAI launched a deployment company with $4 billion behind it. We also had a commercial interest in that outcome, which readers should weigh.

Someone making many loud claims will get some right by chance alone, so a single hit says almost nothing about skill. Without the full set of calls, including the wrong ones, a genuinely good record and a lucky one look identical from outside.

Write the claim down before the outcome, with a date and a number attached, and publish the losses as prominently as the wins. A record put together afterwards from memory is close to worthless, because memory quietly edits out the calls that did not work.

Not in the form it usually appears. It names no date, no measurement, and no definition of "replace", so no outcome can prove it wrong. Rewrite it as a specific claim about job postings falling by a stated amount by a stated date and it becomes gradeable. Far fewer people are willing to say that version.

Read it as commentary rather than foresight, which is a perfectly reasonable thing to read. The mistake is not reading it. The mistake is treating it as insight and letting it shape a roadmap, a hiring plan or a budget.

That by the end of 2027, most production AI work at the companies we serve will run on open models you can download and run yourself, rather than on paid access to a frontier model. We expect frontier models to be kept for a small share of genuinely hard work. If the paid models pull far ahead again, we will have been wrong.

Whenever evidence arrives that would have changed the original call, which is a different trigger from a date in the calendar. The best forecasters update when new information appears. The worst treat sticking to their earlier position as a virtue.