Limited Intelligence

one word, many meanings, trillion dollar bets

A worm turns toward food. A dolphin recognises itself in a mirror. A rat imagines a future it hasn’t lived yet. A toddler dips her hands in paint and smudges the living room wall. A chatbot passes a medical licensing exam. A grandmother says nothing at exactly the right moment.

We call each one of them intelligent.

Intelligent comes from the Latin inter-legere—to choose between, or to read between. At its root, intelligence is just picking: left or right, safe or dangerous, now or later. A worm choosing whether to squiggle toward food is doing exactly what the word originally meant. Somewhere between the worm and the chatbot, we inflated the word beyond recognition.

The word run has over six hundred meanings in the OED. You run a business, run a bath, run a risk. We can handle six hundred variations without breaking a sweat, because we can tell the precise meaning with each usage. Nobody confuses a politician running for election with someone running a fever.

Now let’s try intelligence. A worm steering toward food and a grandmother reading a room. A neural network sorting pixels and a dolphin recognising itself in a mirror. We use one word for all of these—and unlike run, we actually believe they’re the same thing. Maybe different amounts of the same thing, like temperature settings on a dial. Turn it up enough and the worm becomes a grandmother.

And, we’re racing to build a world-changing technology on that assumption, and it is almost certainly wrong.

---

Cardboard Rectangles

Computer scientist Alan Kay once described an experiment. A frog’s brain recognises food as small, moving, oblong shapes. Paralyse real flies and place them in front of it—the frog won’t eat. It will starve surrounded by food. Throw little rectangular pieces of cardboard at it—it will gorge itself sick. The frog sees only what its hardware allows it to see, and assumes that’s the whole world.

Now, of course, we are not like frogs, Kay said. Or are we?

Isaac Newton calculated the motion of heavenly bodies but couldn’t recognise a stock bubble—he lost a fortune in the South Sea crash and reportedly said he could predict the movements of the stars but not the madness of people. Arthur Conan Doyle created the most famous fictional detective ever, then spent his later years attending séances and insisting that fairy photographs were real. Two great minds, each undone by something no test of intelligence would have caught.

Keith Stanovich, the psychologist who spent decades studying this gap, coined a word for this: dysrationalia. The systematic failure to act rationally despite possessing adequate intelligence.

The frog’s hardware is limited. So is the word.

---

straw-man

Adam Mastroianni recently drew a line through this mess. He split intelligence into two kinds. Objective intelligence handles problems with clear boundaries and verifiable answers—math, logic, pattern recognition, the stuff that shows up on tests.

Subjective intelligence is the other kind. A nurse who senses something wrong before the monitor detects it. A comedian timing a pause before the punchline. A negotiator who reads the room and changes strategy mid-sentence. A mother who can differentiate between a tired cry and a hungry one. A poet’s instinct for the line break. A musician’s sense of when to rest.

Hollywood understood this decades ago. It showcased one character for logic and another for wisdom—Captain Spock for objective intelligence, Forrest Gump for subjective—and never tried to cast both in the same role. What screenwriters grasped intuitively, billionaire tech bros are still catching up to.

There’s a Mullah Nasruddin story that captures this perfectly. Every day, Nasruddin crosses the border with a donkey loaded with straw. Every day, the border guards search the straw meticulously. They sift through it, weigh it, sniff it but find nothing. This goes on for years. Long after he had retired, one of the guards runs into Nasruddin and begs him: I know you were smuggling something. I won’t report you. Just tell me—what was it? Nasruddin smiles. Donkeys.

We are like the guards searching in the straw.

---

It’s Not Rocket Science

Helen Lewis has a name for the pattern Newton and Conan Doyle exemplify: the genius myth. The belief that brilliance in one domain transfers automatically to others. A storied engineer assumes that mastery of rocket propulsion translates to mastery of government effectiveness and efficiency.

The pattern is very old. The champions of eugenics were often high-IQ individuals who used their considerable intellect to champion bogus data and pseudoscience with supreme confidence—discrediting their own pretensions to genius in the process. Genius, Lewis argues, should properly be imputed to works—paintings, inventions, proofs—rather than to people. Newton’s Principia was a work of genius. Newton at the stock exchange was not.

In the old world, this delusion was somewhat contained—limited to the genius’s own followers, their own boardroom, in their own era. With LLMs, the stakes are much higher. The biases and blind spots of the people who build and train these systems get baked into the foundation. Applications then get built on top of these models. And users only engage with the model outputs. So nobody sees the cracks underneath, because the surface is so fluent, so confident, so persuasively intelligent that questioning it feels irrational.

Transferable genius is a concept whose time has come—and is being tested, perhaps to destruction.

---

The Orchestra

In my recent essay, Warmer, Colder, I traced what we call intelligence back 600 million years—drawing on neuroscientist Max Bennett's work across five distinct breakthroughs, from a worm's two-rule steering to a fish learning surprise, to a rat rehearsing futures, to a human's capacity for language. Each solved a different problem with different hardware. They aren't points on a dial. They're different instruments in an orchestra. You cannot get from steering to simulating by turning up the volume on steering.

IBM’s DeepBlue mastered one instrument. Deep Mind’s AlphaGo mastered another. We heard a brilliant soloist and called it a symphony. Neither machine understood what it had done, or that it had done anything at all.

The Treachery of Images, 1929 by Rene Magritte

---

42

In The Hitchhiker’s Guide to the Galaxy, a supercomputer spends 7.5 million years calculating the answer to the Ultimate Question of Life, the Universe, and Everything. The answer is 42. When everyone protests, the computer points out that they never actually specified what the question was. We’ve been doing the same thing with intelligence—fine-tuning the answer with more compute, higher scores, better benchmarks—without ever properly defining the question.

Remember my grandmother—the one who said nothing? Imagine replacing her with a chatbot. It would have said something. Something articulate, empathetic, well-sourced, and entirely wrong—because it doesn’t know my family, our history, our loves and hates. I may be smart enough to build the app but not wise enough to know when not to trust its answer.

Run has six hundred meanings and causes no confusion. Intelligence has at least five, possibly dozens, and we’re building a world-changing technology on the assumption there is only one.

The problem isn’t the thing itself. It’s our misuse of the word.

PS: A tip of the hat to Sheo Ratan Agarwal, whose comment on Warmer, Colder pointed me to Alan Kay and Keith Stanovich. Readers make essays better. This one is proof.