In my line of work there is one rule nobody argues with
In my line of work there is one rule nobody argues with: the person who drew it does not get to approve it.
A building going through Building Regulations is checked by someone other than the person who designed it. Not because designers are dishonest, but because whoever wants the project finished on time is the wrong person to say "this part fails".
That is exactly where AI agents are stuck.
On 16 September, Canada and Germany announced up to C$300M between them (C$150M from Canada, €100M from Germany) for LawZero, a non-profit founded by Yoshua Bengio, Turing Award winner and one of the people who built deep learning.
What they are building is called Scientist AI. The official announcement describes an AI that reasons transparently, gives evidence-based answers and has "no goals of its own". LawZero's own site is more direct: its first priority is a guardrail that watches other AI agents. Before an agent acts, it looks, judges whether the action could cause harm, and blocks it if so.
Building Control, for AI.
I think the direction is right, for the same reason as that rule. An agent's job is to finish the task, and the more it wants to finish, the harder it is for it to say "stop". Asking it to check itself is asking the person racing a deadline to sign off their own drawings. The checker has to have no stake in the thing getting done.
Before an agent acts, it looks, judges whether the action could cause harm, and blocks it if so.
The caveats, plainly:
・This is a commitment, not money already paid. Germany's share still needs European Commission clearance.
・Scientist AI does not exist yet. No product, no public results. What exists is a research direction and a budget.
・"No goals of its own" is a design aim, not a demonstrated property.
・"Watching other AIs" is LawZero's stated first step, not a purpose written into either government's announcement.
・Who checks the checker? Building Control has legislation, accountable people and signatures behind it. The AI version has none of that layer yet.
One thing is worth copying today, though, without waiting for the C$300M version: if you use AI agents for real work, split the roles. One does the work. The other only looks for what is wrong, and does not know how badly you want it done.
So my question: for the work you hand to AI now, is there someone, or something, with no deadline of its own taking the last look?
