msvader.net · Research
How AI systems behave inside real workflows. Capability, reliability, failure modes, human oversight, and whether the evidence still holds when conditions change.
Most evaluation is a snapshot. A model is run against a fixed set, a number is reported, and the number gets treated as a property of the model rather than of that particular test, on that particular day, under those particular conditions.
Real systems do not hold still. The material changes. The model changes underneath you. The tooling that measures it gets rebuilt in response to what it found, which means the instrument and the subject are moving at the same time. An assurance that held in March may not hold in September, and nothing announces the moment it stopped being true.
This section is about that problem. Not whether a model can do something, but how you maintain a working understanding of a system that will not stay put.
One lane for accountability. Every lane for sensemaking.
Building sits in the middle of that loop rather than at the end. The artifact is an instrument for observing the system, not the output of the work. "What doesn't quite fit?" is the question that does the real work: anomaly detection under uncertainty is what drives discovery, and it is the one worth automating once the volume exceeds what one person can hold.
Five predictions published in 2021, scored five years on, including the ones that were wrong, followed by a section on what was not seen coming at all.
Systematic investigation Bypassing the MeshFour parts, following one technique across pipeline, budget, operational reality and what comes next.
Failure mode analysis The Translation TaxWhy “AI is a tool” is the wrong metaphor for high-bandwidth minds, and what cognitive symbiosis looks like instead.
A system in which AI carries a large and continuously changing translation workflow, with the boundary of where its output can be trusted established by evidence and re-tested as the material changes.