The difficulty worth keeping when using AI
Desirable difficulties, cognitive offloading, and how to use AI without rotting your brain
✌️ Hey, I’m Caitlin. I help product, design, and insights folks do better customer research with AI—without the hype.
Dive deeper: Claude Code for Customer Insights (June enrolling) | more coming soon
Think first, then let AI finish. Is that enough?
When people in product and research ask me about AI and their brains, the question is almost always about customers: where should I stop using AI so I don’t lose touch with the people I’m building for?
It’s a reasonable question. But it’s not the question I lie awake thinking about.
When I zoom out, I’m far more worried about my own brain decaying across the hundreds of other things I now run on AI. I use it to keep my course content current. I use it to prep keynote presentations, to pressure-test my own thinking, to help me write newsletter editions like this one (usually 80% me, 20% AI).
I’ve been deliberate about it — careful about which tasks I hand over and the order I hand them over in. My have a set of rules, but roughly it goes: think first, let it finish things off.
Over the past few years, I’ve worked out an approach I trust, and the reassuring part is that it lines up with what learning science and decades of automation research already say.
So if you’re the one asking where to draw the line with AI before it dulls you, here’s my answer: what I keep in my own hands, the method I run on everything else, and the evidence for why it works. The goal is to stay in the top tier of AI users — shipping fast, delivering well — while staying sharp enough that you’d still be good at this if someone took the tools away.
In this edition:
🏕️ Why the easy way feels like learning (and isn’t) — the decades-old science of “desirable difficulties,” and why AI is the most convincing illusion of competence ever built
🗺️ Keeping your brain in the chat — how to spot what you shouldn’t hand to AI, plus my blueprint for everything you do
🔭 The view from here — the study everyone’s citing right now, with a skeptic’s hat on
Let’s get into it —
🏕️ BASE CAMP
Why the easy way feels like learning (and isn’t)
There’s a body of research that’s been sitting there, useful, for thirty years and suddenly matters a lot more than it did in 2019.
In the 1990s, the cognitive psychologist Robert Bjork coined the term “desirable difficulties” based on his decades of work with Elizabeth Bjork. The finding, repeated across decades of studies: the conditions that make learning feel slow and effortful — spacing practice out, mixing topics up, testing yourself instead of rereading, generating an answer before you’re shown one — are the conditions that produce the strongest long-term memory.
Those conditions also produce the best transfer — your ability to use the knowledge in a new situation later. In basic terms, the conditions that feel smooth and fast tend to produce the weakest memory and the shakiest transfer — even though, in the moment, they feel like they’re working better.
The catch is in how badly we misjudge this in the moment. If there’s a line from their work to tattoo on the inside of your eyelids, it’s this one:
“Current performance is not a reliable index of learning.”
When something feels fluent — when it comes easily, reads cleanly, slides down without resistance — we read that fluency as understanding. The Bjorks called it the “illusion of comprehension.” Rereading your notes the night before feels productive because it feels familiar. It mostly isn’t doing much. The struggle you avoided was the part that would have built the memory.
The part that should worry every heavy AI user: In one study, people practiced under two conditions, then were asked which one taught them more. The mixed-up, harder condition won decisively on the actual test — roughly 90% of people learned better that way. And yet most of them, even after seeing their own better results, still believed the easy way had taught them more. They were sure they were in the 10% exception.
We are all sure we’re in the 10% exception.
AI is the most fluent thing ever invented. It hands you a clean, confident, finished-looking answer with zero resistance. By the desirable-difficulties logic, that’s not a neutral convenience — it’s the exact formula that produces the illusion of comprehension at industrial scale. The output looks like understanding. Your “current performance” — the deck, the synthesis, the analysis — looks fantastic. Whether you learned anything is a completely separate question, and the fluency is actively hiding the answer.
The early signals
A couple of recent studies are getting passed around as proof we’re all getting dumber. I’d be careful with them — they’re newer and softer than the desirable-difficulties work, and I’ll point you to one below with the caveats. But the direction is consistent.
A 2025 study of 666 people (Gerlich, in the journal Societies) found a significant negative correlation between frequent AI use and critical-thinking scores, statistically explained by “cognitive offloading” — handing your thinking to the tool — and strongest in younger, heavier users. It’s correlational and self-reported, so hold it loosely.
An MIT group wired people up to EEG while they wrote essays with and without ChatGPT. The AI-assisted writers showed the weakest brain connectivity, which the authors called “cognitive debt.” Small sample, has published critics. Suggestive, not entirely settled, but worth consideration.
But you don’t need the new studies to make the call. The thirty-year-old ones already told us: the easy, fluent path feels like learning and isn’t.
AI just made the easy path frictionless and put it on every task you do.
Why it matters: The thing that makes you valuable isn’t the output AI can now generate. It’s the judgment to know whether the task is worth doing, and when that task’s output is wrong — and that judgment is built by exactly the effortful work AI is now offering to take off your plate.
🗺️ THE ROUTE
Keeping your brain in the chat
I want to be honest about the bind, because pretending it away helps no one. I’m under the same pressure you are. We’re all expected to deliver a hundred times what we delivered five years ago, in a fraction of the time. Our world rewards constant action and speed over the slow development of intelligence over time now.
Expertise used to come with time — years of doing the work by hand until the judgment set in. That’s not the deal anymore. You’ve got a training budget you’re supposed to apply in 24 hours, ten skills to learn by Friday, a product to ship by Monday latest. Nobody’s pausing the roadmap so you can mess around for the sake of using your brain.
So here’s the approach I actually run to use AI hard without letting my brain leave the chat. It comes down to two big moves: decide what you’re not going to hand over, then run a real method on everything else. Each piece lines up with research I’ll point to as we go.
Move 1: Figure out what you shouldn’t hand to AI
I’m not going to give you a list of tasks to keep manual — your work isn’t mine, and the right line sits in a different place for everyone. What I can give you is the question I run on anything I’m tempted to fully automate:
Is the difficulty in this task teaching me something I’ll reuse — judgment, pattern recognition, familiarity with my customers or my data — or is it busywork I have to go through to get to the thoughtful part?
Busywork and toil, AI can have it. The difficulty that’s building something in me, I keep, even when keeping it is slower.
Three signals that a task is one of those:
‣ When coverage is negotiable but contact isn’t.
In research the raw material is always the value — so the rule here isn’t “keep it all manual.” You can’t read every transcript by hand, and handling that volume is exactly what AI is for. The discipline is narrower: compromise on coverage, never on contact. Read enough of your own raw data yourself to stay fluent in it, then let AI scale the rest. When I teach analysis with AI or Claude Code, that’s the line I hold — and the counterintuitive part is that if a team is already running heavy AI moderation and analysis, I tell them to add more manual contact back in, not less.
Hand the whole pipeline to agents and you reach “customer understanding” faster — but you’ve pulled yourself out of the understanding. You become the manager of a manager managing bots. Anyone who’s managed people knows how that goes: the further you are from the work, the harder it is to see what’s really happening in it. You lose touch with the customer, and you lose the ability to catch what your agents get wrong, because you’re no longer fluent in the work yourself.
None of this is new, and it’s worth knowing it predates the AI hand-wringing by decades. In 1983 the human-factors researcher Lisanne Bainbridge described what she called the “irony of automation”: hand the routine work to the machine and you leave the human supervising the rare failures — after stripping away the everyday practice that made them able to spot one. You end up least equipped to step in at the exact moment stepping in matters most. She was writing about industrial control rooms. It reads like it was written about your research pipeline last week.
The signal: if you can’t remember the last time you touched your own raw data (or your whole understanding of customer needs come through AI’s summaries), that’s the time to pull back and dig in on your own.
—
‣ When the doing is the thinking — or the relationship.
I don’t automate some whole tasks like email. I don’t have an agent answering for me. Some things take longer to reply to than I’d like, and I’ll take that, because I’d rather respond as a human with my own brain than have an agent transacting with other humans on my behalf while I have no idea what’s actually being said. The signal: if offloading the task would also offload the thought or the human connection that was the point, keep it.
—
‣ When it’s the actual decision.
This is the one I’d defend hardest. The reader I worry about most is the PM who’s been pushed to use AI across so many tasks, so fast, that they can’t keep an eye on any of it — and who, at some point, starts handing off not just the work but the decisions.
Sitting with all the evidence and making the call yourself is slow and hard, and asking the model what it would do is fast and easy. The more we give the decision itself to AI, the harder it gets to make decisions on our own. That capability doesn’t announce its departure. It just stops being there when you reach for it. The signal: if you’re about to ask AI what you should decide, that’s the line. Gather, draft, format with it. Keep the call to yourself.
〰️
Move 2: When I’m using AI across the board, here’s my blueprint
For the work that is worth handing over, I use the same four steps every time:
Think first → take notes → give the notes to AI → let it finish → check it against your own read.
The thinking and the structure stay mine. AI does the assembly and the polish. The step I guard is the first one — when “think first” shrinks to nothing because I’m rushing, I’ve stopped doing the part that keeps me sharp and I’m just approving plausible-looking output.
Reverse the order — let AI generate first, then react to what it made — and you’ve swapped generating (the strongest desirable difficulty there is) for recognizing, which feels just as smart and builds almost nothing.
The last step matters more than it looks. “Think first” isn’t only about staying sharp — it leaves me with my own read to hold the AI’s version against. So “check it against your own read” isn’t passive editing; it’s putting what AI produced next to the answer I already committed to and hunting for where the two split. That gap is where AI either caught something I missed or confidently made something up. Without my own version first, I’ve got nothing to catch it with — which, it turns out, is the same finding researchers keep landing on: the people who engage their own judgment before they see the AI’s answer are the ones who don’t get captured by it.
Here’s what the blueprint actually looks like across the tasks I run with AI in the mix:
This newsletter. I work out the argument, the structure, and my own takes before AI touches it. It drafts and tightens around my thinking and my notes; I edit every line. That’s how a piece lands at 80% me — the reasoning was done before the drafting started. The perspective, the experiences, the stories are already there — and they came from my brain, in my words.
Client presentations. I build the narrative and land on the findings myself — that’s the part clients are paying for. I write the spine in notes, I reflect on previous presentations, then let AI flesh out and polish the content into slides so I don’t have to (I’m not learning anything from slide formatting).
Data analysis. Before I open AI on a batch of transcripts or survey responses, I read enough of it myself to write my own three-bullet read of what’s going on — generated from my own head, badly if necessary. I get a feel for what I think is happening in the data. Then I run analysis with AI and compare: where did it catch something I missed, where did it miss something I caught, where did it confidently make something up. Having my own read first is the only reason I can tell.
Making decisions - “A or B?”. When I’m genuinely torn — a pricing change, which course to build next, whether to take a client on — I write out my own read first: the two options, what actually matters to me here, where I’m leaning and why. Then I hand that to AI and ask it to come at me: poke holes in my reasoning, make the strongest case for the option I didn’t pick, name the tradeoff I’m pretending isn’t there. What I never do is ask it which one to choose. It’s a sparring partner for the thinking, not the thing that makes the call — and yes, this is the same decision line from Move 1, which is exactly why I’m careful about it. The reasoning gets sharper; the choice stays mine.
〰️
AI never does the original thinking for me or makes the final call. It assembles, structures, and polishes what I’ve already reasoned through.
Why it matters: Staying in the top tier of AI users — while using your own brain — isn’t about offloading the most. It’s about offloading the right things and deliberately keeping the few difficulties that are still making you better.
🔭 The view from here
If you want the uncomfortable version of all this, the MIT Media Lab study making the rounds — “Your Brain on ChatGPT” — is worth the twenty minutes, with a skeptic’s hat on: it’s a small-sample study, so read it as a provocation, not a verdict. The sturdier read is anything from the Bjork Learning and Forgetting Lab on desirable difficulties — thirty years of evidence that the easy way has always been the illusion. AI just made the easy way available everywhere at once.
What do you want to read next?
Last thing: I’m always trying to learn from you all about what specifically you’re struggling to figure out and implement in your work. I want this newsletter to be the best thing in your inbox — tell me what you need more of 🫶
—
Keep moving.
— Caitlin Sullivan
Love this! “Some things take longer… and I take that…”