some cranky thoughts about that nyt ai quiz
well really ursula k le guin's place therein
Toward the end of February, Jack Clark of Anthropic recommended three books to listeners of Ezra Klein’s podcast for the New York Times. The first of these was, oddly enough, A Wizard of Earthsea:
Ursula Le Guin’s “A Wizard of Earthsea.” It was the first book I read. It’s a book where magic comes from knowing the true name of things, and it’s also a meditation on hubris—in this case, of a person thinking they can push magic very far.
To the best of my knowledge I’ve never actually listened to an episode of the Ezra Klein podcast. I listen to podcasts almost exclusively for falling asleep purposes and thus mostly listen to old radio shows or true crime. So I did not hear this comment on the podcast. Rather, I became aware of this comment because Le Guin’s son, Theo Downes-Le Guin posted about it on Instagram with the comment:
I am gratified to learn than Jack Clark of Anthropic loves ‘A Wizard of Earthsea,’ and that he sees in it a means for recognizing one’s own hubris. His fellow archmages of AI would do well to follow his lead. This would be more gratifying if we weren’t just now finishing up paperwork for the class action settlement witb Anthropic for training Claude on 40 so of Ursula’s books, using pirated versions and without permission.
I read this Instagram post, gave a dry chuckle, and then forgot about it until the New York Times released a quiz putting up passages from famous works against an AI version generated by Anthropic’s Claude and asked which people preferred.
And one of those passages was from… A Wizard of Earthsea:
These three uses of Le Guin (piracy, interview, quiz) are all from different sources but have the cumulative effect of watching somebody try to get their crush’s attention by shooting spitwads from the corner. Why Le Guin? Sure, she probably is a childhood favorite of Clark’s, but since the quiz also includes Hilary Mantel’s Wolf Hall, that’s not why she’s there. There are many speculative fiction writers who welcome the idea of technological posthumanity—who are even a little bullish on AI-ruled futures—such that having an AI version of their text might be sort of meta, but Le Guin isn’t one of them. You could rewrite a passage about HAL 9000 to be cute and funny.1
Frankly, it is hard not to feel like some of the draw here, for somebody, is that she would have objected to her words being treated this way by this technology: you might have wanted to stay a hundred miles away from this shit, but we can drag you into it anyway. Certainly that would have been true of the other writers selected for the quiz.2
But if the quiz is an exercise in humiliating the writers featured, it’s also about humiliating the journalists making it. Journalists (and writers) as a group are very invested in a self-mythology that believes that being a journalist (or writer) is The Most Important Job Ever, and perhaps for that reason the idea—the idea, not the reality—of LLMs writing anything at all is hitting journalists like rabbits hitting the ecosystem of Australia. There are plenty of reasons to be demoralized about the state of professional writing—I would never say otherwise—but it has to be said that professional writers are hellbent on demoralizing themselves here. Whatever the battle is here, they really want to lose it as far as in advance as possible.
Thus, to me, the ultimate message communicated by this quiz is:
Writing is so worthless that even these so-called masters of the form are easily beaten by an LLM. Weep.
Wow, look at this LLM and the amazing things it can do! It rewrote a paragraph!
So is writing amazing or isn’t it, etc etc etc.
I’m not knee-jerk opposed to LLMs, which seem like a useful assistant technology for some people who already know what they’re doing, and a disaster for every person who doesn’t (like students). They are not useful to me, though Google is becoming such a worse search engine day by day that they might become useful in the future. They couldn’t write my book, which is a different claim from whether or not anybody will read my book, and I don’t mean that as a statement of pride but as a statement of fact. I don’t really have anything to say about “AI art” because… I already said what I had to say in this post from last year. I don’t really worry about “AI” when it comes to myself. What is “AI” going to do to me? Force me to live in my parents’ basement? Big deal, I already did that to myself.
But I do find this particular exercise weird and sad. Have a little self-respect, guys. This thing where you proclaim your own lack of value quarterly isn’t necessary. If you really feel that way, go get a nursing degree or something.3 Anyway these are only short and cranky thoughts, take them for what they’re worth, which is a way to spend some time drinking a third of a cup of coffee. (Which is about where I am.)
Also I think Ursula K. Le Guin’s ghost should be allowed to wreck some data centers.
Surely there’s a chatbot named HAL.
With the possible exception of Carl Sagan. Dunno about him.
The unspoken belief I sense over and over is “computers were supposed to make everybody else redundant, but not us.” Maybe worth wondering why you were so happy with that bargain on your way to the panic store.
i really resisted taking this quiz when i saw it on twitter but eventually i caved in and the premise was so... much weirder than i thought? originally i thought the test was trying to force people to judge between a famous writer's writing and passages that claude had generated from an original prompt -- such that people would have to confront whether claude can produce "better writing" than human writers -- but every text was so blatantly just rewriting exactly the original passage (but more contrived and heavy-handed)
and to me that framework just undercut the stakes of llm writing (a problem that this test seems to be trying to confront)... like yes i could also copy a passage from a hilary mantel book and play thesaurus games with it... why would i feel threatened by that
What you said about the writer/journalist freakout about AI reminds me of how writers talk about 'slop'. There's this consensus that AI writing is 'slop' and bad and essentially meaningless junk, which I agree with. But writers have been producing junk for ages (no offense!), especially on digital platforms (and before!). Like, writers and journalists are talking a big game about how what they do is the most meaningful vocation that has ever existed, and how AI will be the death of writing, but then these writers go off and post listicles and twee roundups and shopping recs. It's not a nice conversation to have, but I think if people are actually going to be honest about AI and the proliferation of bad writing, we also need to talk about how actual writers create slop just well enough on their own. Just a thought!!