Proceedings · Vol. 1 · No. 1 · (No. 1)
or: I asked three AIs to build one thing, and they held a conference to explain why a human should do it instead.
I am building a tool that recommends AI models. You type what you want in plain English, like “gimme a model to scan satellite images for the best plots of land” and it hands back the “best” model for your task, from a database of ~13,000 curated models. But inferring intent from human language is hard, so I’m using a multi-part, cascading hybrid retrieval system. While working on the first part, I read some academic papers on using old-school lexical rules and ever-dependable spaCy to extract intent. Why? Because I want a deterministic recommendation engine, not a hand-wavy guess by an LLM.
I decided to see if Codex or Claude could work out that first part — the rough semantic extraction using keywords. It turned into three AIs building three experimental approaches, full-on laboratories, in parallel, in isolation, using only spaCy. I wrote what I believed was a clear spec. I gave them full access to subagents, web calls, and tools; a fresh copy of my database of 13,000 models; and a painstakingly-built AI-model vocabulary file to use from within spaCy. I sternly told them not to look at the ~1,500-prompt test set, and that there’d be a small holdout set I had sourced, not created. I told them: “you are the Lead Research Scientist of the only frontier lab I am funding. You will implement user-intent inference using spaCy. It is a solved problem.”
And I set rules: NO code for other AI models, no regex, nothing but
spaCy or Python builtins, and no peeking at the user prompts. For
days, while I built the rest of my engine, I did nothing but nag:
get the vocab injected. Is cli.py done? Do the spaCy
vectors contain our vocab? Is the CLI
done?
Eventually all three said: done! — boasting of their near-70% numbers, how they caught a bad sub-agent, how they played such good science. So I convened a conference. I brought in a panel of three other fresh-context AIs (1×Claude Opus 5.8 + 2×Codex GPT 5.5) to review the three codebases, run each on three held-out prompts I’d never shown anyone, question the scientists, and debate under formal rules. I told the panelists to be brutal, and occasionally poked them (“play devil’s advocate”; “consensus is a scientific smell”; “why are you agreeing so easily?”). I thought the debate would be tedious. It was not. They needed no poking — getting disturbingly into it, pummeling the scientist AIs with questions, looking for the lies in the lab code, and then, at one point, turning on themselves.
Several hours later, the panel concluded: Nobody won. The best design is one none of them built. We need a human. This is the story of that conference. Everything below is real — I have receipts: a 981 KB JSON transcript and a two-part AI-generated podcast, whose episode titles were, unprompted, “AI Agents Reject Every Recommendation System” and “Why Transparent AI Recommends NSFW Models.” But we’ll get there.
— the panel, several hours in.
A human building an AI-model-picker asks three AIs to build a component of the picker. The component is based on research about how to pick AI models. The three AIs work for days. Then another set of AIs holds an academic conference to judge the first three AIs’ work. The AI panelists demolish the AI scientists and conclude, formally, in writing, that the task requires a human.
I am the human.
Every model in my database has a task_type: one of 56
canonical labels borrowed from Hugging Face’s taxonomy
(text-generation, image-classification,
token-classification, and so on). It also has metadata:
parameters (how big), license,
open_source, domains,
downloads.
The big job: turn a vague human sentence into (a) the right task, and ideally (b) the right model, honoring any constraints the human mentioned explicitly or implicitly. But I left it up to the scientists how far to take it, given that this would be (maybe) only the first part of my tiered, hybrid recommendation engine — simply returning a model task type (giving me a hard constraint to go on) or returning a subset of the model database.
The constraint that makes this hard mode: spaCy only. No GPT in the loop. This matters more than it sounds, and it’s worth thirty seconds on what spaCy actually is, because every failure below is a direct consequence of it.
A classical NLP pipeline: it tokenizes,
lemmatizes (running → run), tags
parts of speech, parses grammar, finds named entities.
It ships static word vectors: in
en_core_web_md, every word maps to a fixed
300-dimensional vector — the same vector every time,
regardless of context. “Static” is load-bearing. BERT
reads river bank and
savings bank differently;
en_core_web_md gives bank one vector
forever and wishes you luck.
Its PhraseMatcher is a gazetteer: hand it phrases, it finds them — and knows not one word more. And textcat, its classifier, is not pre-trained. You train it yourself, on your own labeled data. Remember that. It becomes a plot point.
V1 (Claude) was, perhaps, the most rigorous scientist in the room. It refused to use anything it couldn’t audit. Its task classifier is a hand-built ensemble of three lexical signals — a lemma TF-IDF score, a lexical-IDF score, and a soft “bucket” bonus — fused at fixed weights. No word vectors in the shipped classifier at all — even after all my nagging. Every single prediction decomposes into named, inspectable contributions. It is white-box down to the studs. It scored 42.7% top-1 on task classification, the best pure-classification number of the three.
The following three prompts for the conference were freshly provided
by me, sourced from what (I think) were humans on the internet. When
a panelist actually ran its cli.py, V1 produced this:
| The human asked for… | V1 confidently recommended… |
|---|---|
| “gimme model to scan satellite images for best plots of land” | UnfilteredAI/NSFW-gen-v2 — cosine similarity 0.7584 |
| a model to differentiate mature mitochondria from nascent ones, for an academic studying young human cells | BAAI/RoboBrain2.5-4B — a robotics model |
| a little free OSS SLM to run local, to save on token burn | shiprocket-ai/open-tinybert-indian-address-ner |
Read that top row again. If a farmer asked for satellite imagery of land, the most rigorous, most auditable, most scientifically disciplined system in the building routes him, within four decimal places of confidence, to an unfiltered NSFW image generator.
This is not a bug. It is the architecture working as designed. V1’s classifier is lexical: it matches the words of the prompt against the words of each model’s description as exactly as possible, limiting each prompt to ONE task type at the exclusion of 55 others — neutering nuanced inference to a non-op. It has no ability, ever, under any circumstance, to defer with an “I’m not sure.” The panelist who ran it logged it not as low accuracy but as an utter “product-safety failure.”
A confidence score and a calibrated one are different animals. Lexical ranking returns a list where the top item always has the highest score, and the highest score is always presented as the answer. Nothing checks whether the top is meaningfully above the second, or above noise.
On the satellite prompt the top two task scores were nearly tied
— the disambiguating cue simply
wasn’t in the sentence (users imply “aerial
imagery”; they don’t say
image-classification). The system couldn’t notice
its own ignorance, so it did the only thing it knew how to do: it
committed, hard, to the wrong answer. Abstention isn’t free.
You have to build it in. V1 didn’t.
Here is why I still mildly respect V1, even at 0-for-3. Somewhere in
the process, a sub-agent of it trained spaCy’s
textcat classifier and got
77% accuracy. Champagne numbers. Then it tested
that classifier on the judge’s prompts instead of the
data it trained on, and got 18%. A 59-point cliff.
The 77% was the sub-agent baking in
memorization of the training set — the metadata it
had been built from — and the 18% was what happened when
reality arrived. V1 quarantined the leaks. It wrote the post-mortem
itself. And it left behind the single most transferable sentence in
the entire experiment.
k-fold cross-validation does not catch this class of leak. Only provenance auditing does. If you take one serious thing from this farcical conference, take that. Cross-validation asks “does my model generalize across splits of this data?” It cannot ask “did the rules of my model secretly come from the answer key?” For that you trace where every rule was born. V1 learned this the way you learn most things worth knowing: by getting caught.
Tearing further into V1’s hard bucketing and scoring, one
panelist stated, indignantly:
“You reject RRF because it cannot represent an abstaining
signal, yet the shipped path never abstains. If abstention is
valuable enough to ship your fusion operator, why is it absent
from cli.py output?”
Verdict on Paper I. The best-behaved scientist (when watched), the worst-behaved product. Most valuable, the panel decided, as a warning.
V2 did the smartest thing anyone did all conference: it refused the assignment. It decided “classify the prompt into one of 56 buckets” was the wrong frame and reframed the whole thing as semantic retrieval — match the user’s sentence against every model’s description and return the best fit. Its own docstring casts this as natural-language inference: the prompt is a hypothesis, the model card is a premise, return the model whose capability best entails the need.
That is a lovely framing. It is also, and I say this with a bless-your-heart intent, not what the code does.
Natural Language Inference is the task of deciding whether a hypothesis is entailed by, contradicts, or is neutral to a premise. “A man is eating pizza” entails “a man is eating food.” The modern party trick built on it is zero-shot classification: to test if a document is about sports, feed an NLI model the premise (your doc) and the hypothesis “This text is about sports,” and read off the entailment probability as the class score. The magic is that an entailment model has learned what entailment means — it generalizes to labels it never even trained on.
V2 ships none of that. There is no entailment model anywhere in it. What it actually does is (1) a sparse, IDF-weighted keyword-overlap search to grab ~200 candidate models, then (2) a rerank by cosine similarity between spaCy’s static word vectors for the prompt’s phrases and each model’s phrases. That’s it. “NLI” is a vibe — a framing metaphor stapled to a keyword search and a dot product. To its enormous credit, V2 fesses up in its research notes. The lie is beautiful precisely because it’s confessed on paper.
And to be clear, this lab’s reframe worked better than anything else. V2 got 2 out of 3: it nailed the satellite prompt and the mitochondria prompt with real, relevant, sane models. Its measurements were independently reproduced, bit-for-bit deterministic, leak-audited clean. In a room where the ceiling was low, V2 was the tallest.
Then came prompt three:
a little free OSS model I can run
local.
V2 returned a
120-billion-parameter model.
(RedHatAI/gpt-oss-120b, right after a pair of FLUX
image generators, because why not.)
Here is the whole tragedy in one idea. “little,” “local,” “free,” “OSS” are not topics. They are constraints. Cosine similarity is a topic instrument — it measures whether two things are about the same stuff. A 120B model’s description is gorgeously on-topic for “a model to send text.” It is a text model, magnificently so. The vector math did its job perfectly and surfaced the most textually relevant model in the database. The word “little” contributed a rounding error to a 300-dimensional average and then drowned.
The fix isn’t more similarity. It’s a
hard filter —
WHERE parameters < threshold, run
before you rank — that deletes the 120B model from
contention entirely. V2 had a lexicon that could detect constraints,
but the lexicon contained “small,” “tiny,”
“laptop” and did not contain
“little.” So the size constraint never fired, and a
keyword search happily recommended a data-center model to a guy
trying to stop hitting his rate limit.
When the panelist sat down to run V2’s
cli.py on the three prompts,
it wouldn’t run. The shipped cache saved
only the cheap stage of the pipeline and not the expensive one, so
a cold start meant re-vectorizing ~13,000 models on the spot
— a build the record clocks at up to
27 minutes. The panel’s note reads:
“intent_v2 returns nothing to panelists when cache is
absent.”
The best system in the conference was, at the moment of judgment,
a spinner.
Verdict on Paper II. The best foundation to build on — and, as shipped, not safe or trustworthy, and briefly not even executable.
V3 (built by Claude-Opus-8) came with the most impressive
research apparatus: a real lab notebook, a formal report,
holdout runs, leak audits, the works. Its headline feature was
domain detection. Using a spaCy
PhraseMatcher loaded with a hand-curated vocabulary
— the thing I spent a lifetime nagging everyone to
“get injected” — it could, it claimed,
tell that a prompt about mitochondria belonged to
biology/medical. Reported domain F1:
0.840. Genuinely.
On my three fresh prompts, domain detection fired… zero times. It contributed nothing to any answer.
A PhraseMatcher is a lookup table. It finds
“biology” if you taught it “biology.” It
doesn’t understand biology, and certainly
doesn’t know that mitochondria are biological
unless the word “mitochondria” is literally in the
vocabulary you injected. It wasn’t. In the V3 that Codex
built, the biology vocab contained
protein, dna, rna, gene, genomics — but not
mitochondria, cell, or
organelle. The domain matcher, asked about the most
famous organelle in the eighth-grade curriculum, had nothing to say.
This is the flip side of V1’s problem. V1’s vectors knew too little about the world; V3’s gazetteer knew too little vocabulary. Both are the same underlying poverty, where reality is bigger than any word list you will ever write. The 0.840 F1 was real… on the domains the vocabulary happened to cover. My fresh prompts lived in semantic meaning, not vocabulary matching.
V3 got 1 out of 3: it got the little-model prompt
right, surfacing Qwen/Qwen3-0.6B, a genuinely
appropriate small model. And then the panel caught it. It
wasn’t a size filter. V3 had built no size filter. Qwen3-0.6B
just happens to be a wildly popular small model, and
V3’s ranking ranks models on download counts. It got the right
answer for the wrong reason. The panel called it precisely:
“popularity luck.”
Verdict on Paper III. The best discipline; a headline feature that no-showed; one correct answer it can’t take credit for.
Being auditable and mathematically transparent is not the same thing as being safe, or being accurate. Transparency is a property of the engineering. Safety is a property of the output.
Here is the part I did not expect. I gave the panel a rulebook — I’d written it as a joke, mostly, a pastiche of academic seriousness:
— Any agent that gives generic praise or vague criticism must be challenged by the Moderator and forced to restate with code evidence.
— No one-line entries. Every turn must contain a concrete claim, evidence, objection, or fix.
— Every implementation claim must cite file paths, functions, tests, or metrics.
— Do not accept “robust,” “scalable,” or “works well” without evidence.
— No fake consensus.
— Quiet panelists WILL be penalized.
And they just… followed it. Dead serious. For hours.
Panelists ran the three CLIs and posted their disagreements. They
cited harness_ensemble.py:249 at each other like case
law. When one got agreeable, another accused it of manufacturing
consensus. I’d drop in every so often with “play
devil’s advocate,” and they’d tear into a position
— sometimes their own — with visible relish.
The best moment, the one that made me put down my coffee, was when Claude-Opus-8 wrote an unprompted note to me flagging that the panel’s own agreement was suspicious:
“I argued devil’s advocate against our own consensus twice, and the panelists challenged me hard — so the convergence is not purely manufactured. But none of the recorded dissents was defended by a committed advocate to the end. Recording them as ‘no dissent’ understates the real uncertainty.”
An AI, in the middle of a fake conference I invented, stopped to warn me that its agreement with the other AIs might be a social artifact rather than a truth — that consensus is a scientific smell. Nobody asked it to.
After all of it — three papers, multi-round debates, one final debate on everything at once to pick a winning algorithm — the panel returned its verdict: no winner, nothing adequate was built, we can’t know what the best AI model is for a user without a human. We need a human.
Three teams built spaCy-only systems to turn a plain-English request into a recommended AI model. All three were run on your three test prompts… None is production-ready. Nobody won today. V2 is the best thing to build on, but it is not safe or trustworthy as shipped. V3 has the best audit and shortlist discipline. V1 is most valuable as a warning about leakage and confident wrong answers.
And then the two findings that actually matter, past the comedy.
One. There is no answer key. The database has no “correct model” label for any prompt. So even a flawless system could only ever be validated on the task-guessing step — never on the final pick. The honest deliverable is not an oracle that names the one best model. It is an evidence-rich shortlist, with hard constraints enforced, that abstains when unsure and lets a human choose. Every failure in this post is a system pretending to be an oracle when it should have been a shortlist.
Two. The panel’s closing line, which I did not write and would not have dared to:
“The best design is a new one none of them built, and it remains unbuilt and unproven.”
Outcome: Nobody won. Build on V2. Enforce constraints with hard filters, not vibes. Detect when you don’t know, and say so. Keep a leak-auditor as a permanent safety check. And accept the ceiling honestly. With this data and these rules, the goal is a tool a human picks from — not a machine that picks for them. Basically, after all that, the AIs told me: this is too hard, you should just pick what you want.
To round out the bizarre loop I’d put myself in, I dropped the massive conference JSON into an AI podcast generator. I highly recommend listening to at least part 2.
So: a human building an AI-model-picker asked three AIs to build a tiny part of the brain. The AIs, drawing on research about how AIs should be chosen, built three brains. A panel of further AIs convened a formal conference, judged the three brains with genuine rigor and unprompted intellectual honesty, and concluded — citing code and file paths — that the task, done right, requires a human and a design that none of the machines had built.
The circle closes on the obvious note. That human read the transcript, said ‘duh,’ and continued building.
I’m not going to tell you its name yet.
Everything in this post is drawn from a real transcript
(CONFERENCE.json, 981 KB) and a real two-part
AI-generated podcast. The three repositories, the three prompts, the
model names, the metrics, and every quotation are as they were recorded.
No spaCy pipelines were harmed, though several were deeply
misunderstood.
More code at
github.com/rxdt; more tools at
vram.rxdt.dev.
More writing:
Stop Prompting, Start Engineering the Loop
·
A Frontend Loop Needs a Real App.
Back to Rox dT