The test that passes because you chose the wrong examples

What we were doing, so this makes sense

The background. Our site used to live on a website builder, the kind where you drag things around in a browser and somebody else owns the machine underneath. That is a reasonable way to start and a bad way to continue. We were turning a prototype into a real product, and a product needs a site we actually control: our own domain, our own code, our own pipeline, no monthly fee for permission to publish, and nothing that can change its terms on a Tuesday and take our pages with it.

So we moved it. Out of the builder, into a static site we build ourselves and serve from our own storage.

And we did the move the way we do everything here, which is the thing this company is actually about: working alongside an AI assistant. Not watching it work. Not handing it the keys. Working with it, the way you would work with a very fast colleague who has never seen your business before.

That is what we teach, and it is what this post is about, because the interesting part of working this way is not the speed. It is knowing where it goes wrong.

A green test is evidence about the cases you thought of. It is evidence about nothing else.

What we needed that day

When a site is a set of files in storage rather than a program running on a server, something has to translate between what a reader asks for and what is actually on the disk.

A reader asks for /about. The storage holds /about/index.html. Something small sits in the middle and rewrites one into the other.

We also wanted /ConfirmEmail and /confirmemail to be the same page rather than two addresses for one thing, so that same small piece of code lowercased the address on its way through.

It was written, it was tested, every case passed, and it was reported back as working. Here are the cases it was tested against.

/ /about /ConfirmEmail /sitemap.xml

Look at those for a moment before we tell you what happened.

Every one of them is either already lowercase, or so plainly a page address that lowercasing it does no harm. Four examples, all agreeing with each other, all written from the same assumption as the function, all confirming what that assumption already said.

What the build actually produces

The build does not write /about. Among other things it writes this.

/assets/index-D4kM9xRt.js

Those letters after the dash are not decoration. They are a fingerprint of the file's own contents, which is what allows a browser to cache that file forever and still pick up a new one the moment anything changes. Change one character in the source and the name changes with it.

And a fingerprint has capital letters in it, because it is case sensitive. So is every key in the storage bucket.

So the function lowercased it into index-d4km9xrt.js, which is a file that has never existed anywhere.

The site went live serving every page with no stylesheet and no JavaScript. Pages of unstyled black text on a white background.

How we found it, which is the actual lesson

Not with the test. The test was green and stayed green.

We found it with a gate: a check that runs against the finished build, and then again against the live site a minute after publishing, and that blocks the release rather than printing a warning.

It caught it on the first run. Not because it was clever and not because anybody had anticipated this particular mistake. It caught it because it was looking at the real output instead of at an example, and the real output was full of capital letters.

How we fixed it, and why this is the efficient way

The fix inside the function was one line. That part is not interesting and copying it will not help you.

The fix in how we work was three rules, and those are worth stealing.

One. Test against artefacts the system produced, not examples you typed. If a function handles filenames, take the filenames out of the last build. They are sitting in a folder on your disk. Reading four real ones takes less time than inventing four fake ones, and it is the difference between proving something and agreeing with yourself.

Two. Put something in the pipeline that has no opinion. A check that runs on the built output and again on the live site, and that refuses to publish when it fails. Not a warning. A warning is a thing you scroll past at eleven at night when you want to go to bed.

Three. Treat a passing test as a question. The question is: what did this actually prove, and what was it structurally incapable of proving? Four lowercase paths proved that lowercase paths work. Nothing more.

Why this is more efficient rather than more careful: a gate costs you about twenty minutes once, and then it costs you nothing. Finding this by hand costs you a live site, an evening, and the part where you do not know what broke.

The honest part, and it is not about who typed it

The obvious version of this story is "the AI wrote a bad function". That is not what happened and it is not a useful lesson.

Here, as always, it was a collaboration. Some days the assistant writes the code. Some days we do. On this one it does not matter, and we are not going to pretend it does, because the mistake would have been identical either way.

What actually happened is this. The function was written from an assumption about what addresses look like. The test examples were written from the same assumption. And then the review was done by somebody holding that same assumption, which is why the review passed.

That is the failure mode, and you will notice it has nothing to do with who held the keyboard.

Three things agreed with each other. None of them had looked at the real output.

The human layer is not optional, but it is not sufficient either

It does not matter who writes the code. A human reviews the result. That is the whole discipline of working with AI and we do not skip it on any day of the week.

But this is the part people miss, and it cost us a live site: a human reviewing against the same assumption the code came from adds nothing. You read it, it matches what you expect, you approve it. You have confirmed your own belief at the speed of reading.

So the human layer has to be pointed at the right thing. Not "does this look correct", which is an opinion. Does this match what the system actually produced, which is a fact you can go and look at.

That is a different job, and it is the one worth your attention. Reading code for plausibility is the cheap half. Checking it against reality is the half that catches things.

If you are building anything with AI right now

This is the failure mode to be frightened of. Not nonsense. Confidence.

You will be shown a function, an explanation, and a list of passing cases, and all three will agree with each other, because all three came from the same place. Nothing in that chain has touched your actual system.

Working well with an assistant is not about writing better instructions. It is about making sure that something in the room is looking at what really came out, instead of at what everybody expected to come out.

That is the whole skill. Everything else is typing.