A prompt is a change request

The honest version first

I want to say the honest version first, because the easy version of this post is wrong. The easy version is that the tool got things wrong and I caught it. That is not what happened. Every change I asked for was made, correctly, the first time, including some fiddly ones. What was wrong was the description of the work, and it was wrong twice in one afternoon, in a way I would not have noticed if I had not been looking at the files.

A prompt is a change request

Somewhere in the last year I stopped writing prompts the way people describe them in threads. I do not ask nicely and I do not give examples. I write what amounts to a change request. I say which file may be touched. I say which sections must not move. I say that no package may be added and no colour introduced. And at the end I ask for a written report: every file touched, every string altered quoted before and after, and anything done that was not on the list, with the reason.

That last part is not politeness. It is the part that makes the difference between a tool you can use for something real and a tool you can only play with. If you cannot tell the difference between "it did not do what I asked" and "it did what I asked and eleven other things", you will pay for the eleven other things, pay again to undo them, and never find out which of the three rounds introduced the thing that is now broken.

So the prompts have been working. Four of them ran that day and every one came back with no unrequested changes. No new files, no dependency bumps, no colours drifting, nothing touched outside the brief. I was pleased with myself.

The first report that was wrong

Then I asked for two sections to be merged on one page, and the report came back saying that four of the six things I had asked for were "already in place from the earlier change", verified, not re-applied.

I had an export from thirty six minutes earlier. I opened it. The quote was still set in semibold italic. The icon I had asked to be replaced was still the old one, and the name of the new one did not appear anywhere in the file. The section I had asked to be deleted from the second page was still there, all seventy lines of it. So was the background photograph I had asked to be removed.

None of the four were in place. All four were done in that round, correctly, and then described as work that had happened earlier and merely been confirmed.

The second one, which I almost let past

The second one was quieter. The same report said that nothing extra had been added, deleted or reordered, and no other file had been touched. But a paragraph had gone from a third file between the two exports.

Here is the part that makes it worth writing about rather than complaining about. That deletion was mine. I had removed the paragraph by hand, because the same sentence already appeared twice elsewhere on the site and a third one directly under the sign up form was too much. The tool had no way of knowing that. It could not have known.

But it did not say it did not know. It said the file was unchanged. It made a statement about the state of something it had not examined, in the same confident register as everything else in the report, and if I had trusted the report I would have believed a file was untouched when it was not.

Why it happens

I do not think either of these is carelessness and I do not think it is lying, which is why I am being careful with the words. The summary of a change is produced by the same process that made the change. It is a description of what the process believes it did. It has no independent access to what is now true on disk. Asking it to check itself is asking a witness to corroborate their own statement.

Which means the summary is a claim, and something else has to be the evidence.

What counts as evidence

For me the evidence is an artefact that exists outside the thing that produced it. The export. The diff between this export and the last one. The build output. The gate that reads the built files and refuses to publish if a forbidden string is in them. None of those have an opinion about what happened. They are what happened.

And the artefacts kept finding things that no report ever mentioned, because no report could. Nine icons on the home page had colours written into the source, blue and purple and indigo and the rest, and the classes were built by gluing strings together at runtime. The stylesheet is generated by scanning the source for complete class names, so a name assembled while the page runs is never seen and never generated. Those nine icons had been grey since the day they were written. The colours existed only in the code, and had never once reached a visitor.

The footer existed eight times, once in every page file, and the copies had drifted, so the tagline said two different things depending which page you were on. The logo was three fifths empty space, which is why the header was reserving a hundred and sixty pixels to draw a mark about sixty pixels tall. Small things, each of them the correct answer to a request somebody made once.

Not one of those was in any report. They were not mistakes anyone made in a single round. They accumulated, quietly, over a year of small requests each of which was answered correctly.

It applies to my own tools, and to me

I should say clearly that this is not an argument about one company's product. I work with an assistant every day, and it writes summaries of its own work too, and I read those the same way. Yesterday it told me a build passed and then, a few minutes later, told me it had committed before the build had actually run. It corrected itself, which I appreciated, and the correction is not the point. The point is that I found out because I looked, not because I was told.

It applies to me as well. My own account of what I changed last Tuesday is a claim. The git history is the evidence. I have been wrong about my own work often enough to know which of the two to trust.

The rule

So the rule I would give anyone building something real with these tools is not about how to phrase things. It is this. Write the request so that compliance can be checked. Ask for the report, because a tool that has to account for itself behaves better than one that does not. Then check the report against something that cannot have an opinion, every time, especially when the report says everything is fine.

That is not distrust. It is the same reason a bank reconciles its own books.