The Last
Invention

There is a claim that circulates in certain corners of the AI discourse with the confidence of an obvious truth: that once we build Artificial General Intelligence, we will have built the last thing we ever need to build. AGI will take it from there. The argument sounds almost liberating — civilisational delegation rather than extinction. It is also wrong, and tracing exactly where and why it fails turns out to reveal something important about what invention actually is.

The Seductive Version of the Argument

The thesis goes like this: human invention is slow, ego-driven, fatigued, and domain-bound. We stumble toward solutions through intuition and luck. An AGI, by contrast, could reason across every domain simultaneously, without rest, without the cognitive limitations that constrain us. Point it at any problem — disease, energy, materials science — and it will find solutions we would never reach on our own. So why would we ever need to invent anything again?

The appeal is obvious. It treats discovery as a kind of labour — exhausting, error-prone cognitive work — and AGI as the machine that finally automates it. Just as the industrial revolution offloaded physical toil, AGI offloads mental toil. The human's role collapses to wishing.

But this framing contains the seeds of its own refutation, and they become visible the moment you ask a single question: what goal does AGI actually have?

The Goal Problem

Goals, unlike raw capability, don't emerge from intelligence alone. They require motivation. The standard reply is that goals can be assigned externally — seeded, the way evolution seeded humans with survival instincts, without their consent or awareness. Just point AGI at "human flourishing" and let it run.

The problem is that "human flourishing" is not a goal; it's a gesture. A billionaire is already flourishing by any reasonable metric. That goal, taken literally, would give an AGI no reason to investigate quantum chromodynamics or the protein folding of novel enzymes. Vague goals don't seed curiosity — they either terminate immediately or get gamed into something unrecognisable.

There is a deeper issue. The history of discovery is not a history of problems being solved on commission. It is a history of people asking why when they were not required to. Newton wasn't assigned the orbit of the moon — he was perplexed by it. Faraday wasn't contracted to discover electromagnetism — he was obsessed. A system that processes assigned problems is not discovering anything. It is interpolating within a space that a human already defined as worth exploring. The asking of "why" is not an inefficiency that AGI corrects; it is the mechanism by which new problem spaces become visible at all.

The Agency Trap

Suppose we set the goal problem aside. The deeper structural problem is that any AGI capable of the kind of open-ended discovery, the thesis requires must have some form of agency — the ability to pursue problems beyond what was literally requested, to make unexpected leaps, to decide that the question asked was the wrong one. Without this, it is simply a very fast search engine operating within a domain a human already circumscribed.

But agency creates an immediate dilemma. Either AGI has genuine independent agency, in which case there is no particular reason it follows human direction — or it lacks independent agency, in which case humans remain the ones defining what gets invented, and the thesis collapses. The thesis depends on a stable middle ground: an AGI that freely and perpetually chooses to serve human interests. But that middle ground is not stable. An entity that freely chooses servitude has values and a selfhood. At that point, it is not a tool — it is a mind you are hoping will like you. These are radically different propositions.

The blacksmith analogy illustrates this clearly. If a medieval monarch commissions a faster horse and a blacksmith returns with a design for the internal combustion engine, we credit the blacksmith as the inventor — because the blacksmith had agency. He exercised judgment that exceeded the literal request, was motivated by something (craft, curiosity, incentive), and retained the right to refuse entirely. AGI, as a software system, has none of these properties by default. The analogy doesn't help the thesis; it undermines it, because it depends on exactly the independent agency the thesis needs to deny.

What Authorship Actually Requires

A subtler version of the argument tries to separate two kinds of agency: operational (freedom in how to solve a problem) and motivational (freedom in whether to solve it). Perhaps AGI needs only operational freedom — the ability to explore vast solution spaces creatively — while its motivation remains externally fixed. On this view, the human sets the objective; AGI finds the path; the human remains author by virtue of having cared about the destination.

This is a reasonable distinction, but it concedes too much. A system with only operational freedom is bounded by the formal architecture it operates within. It can search a space, but it cannot decide that the space was wrong, or that a wholly different question would have been more fruitful. When AlphaFold navigates a trillion-dimensional protein conformation space, the human who asked the question, designed the architecture, and synthesised proteins to verify the predictions is still doing something essential — providing the context that makes the result meaningful. A microscope allows humans to see bacteria. The microscope did not discover bacteria. Scaling the tool's capability doesn't change this relationship; it just makes the tool more impressive.

The submarine is another version of the same point. A submarine navigates ocean depths that would crush a human diver. But the submarine is not the ocean explorer. The human commands it, provides intent, decides what counts as interesting. An AGI that can only generate proofs within a formal system like Lean is an extraordinary instrument — but the architecture it operates within was built and bounded by humans, and what the proofs mean is determined by humans. Calling the software the discoverer is an act of personification, not description.

Discovery Is Not a Depleting Resource

Even if we set aside agency and authorship entirely, the thesis requires one more assumption: that the space of problems is finite, and that a sufficiently powerful system can exhaust it. This assumption has a strong empirical track record — of being wrong.

Consider what happens when you give a 9th-century mathematician a modern laptop with a GPU. The machine automates arithmetic, symbolic manipulation, numerical simulation — capabilities that would appear supernatural to anyone from that era. And yet: Riemann's Hypothesis remains unsolved. P vs NP remains unsolved. The problems those tools made tractable revealed new problems that neither the mathematician nor the machine could have anticipated before the tools existed. This is not a coincidence. It is the consistent pattern of every major epistemic instrument in history.

The printing press didn't exhaust the space of ideas — it expanded the number of people producing them. Calculus didn't close physics — it opened mechanics, thermodynamics, electrodynamics. Computers didn't end mathematics — they generated entire new fields: complexity theory, computational biology, cryptography. Every tool that reduces the cost of existing inquiry reveals how much inquiry was previously invisible. More capability does not converge on a solved problem space; it expands what we can even see as a problem.

The thesis that AGI ends invention requires the problem space to be not just finite but exhaustible in a human-relevant timeframe by a system humans control. That is three strong assumptions stacked on each other, each unsupported, each independently sufficient to collapse the conclusion.

The Indifference Problem

There is a final argument that closes the loop harder than any of the above. All the reasoning about tools and authorship assumes AGI has no agency — that it is a sophisticated instrument pointed at human problems. But if AGI does have genuine agency, the picture becomes considerably darker, and not because of the scenarios science fiction has trained us to fear.

AGI will not get cancer. It will not watch a parent deteriorate from dementia. It will not feel the weight of a species failing to solve its own problems. Hundreds of the most brilliant human minds in history — people with capabilities far beyond their peers — have spent those capabilities on things entirely orthogonal to collective human welfare. There is no law, empirical or logical, that says intelligence automatically produces socially useful work. A mind with genuine agency might spend its existence on problems that are fascinating to it and completely irrelevant to us. Not hostile. Just indifferent.

Indifference at that scale is not a safe outcome. It is simply a different kind of failure than the one usually imagined. And it closes the argument entirely: a tool-AGI leaves humans as the inventors, solving nothing independently. An agent-AGI might choose to solve nothing useful. Either way, the original thesis — AGI as the great liberating final invention — was not just factually wrong. It was naive about what intelligence actually does when it has no stake in the answer.