A 403 is not always a refusal

What the 403 was actually saying

The site is files in private object storage, served through a content network. The storage is configured the way it should be: it refuses everyone except the one distribution allowed to read from it, and that distribution may fetch a named object and nothing else. It cannot list what is inside.

So when a request arrives for a page that does not exist, the storage has a decision to make. It knows the object is not there. But the caller is not permitted to know what is or is not in the bucket, and answering "not found" would tell them exactly that. Ask for a thousand names, collect the answers, and you have mapped a private bucket without ever being allowed to read a byte of it.

So it answers 403 for both cases. The honest translation is not "you may not". It is: there is nothing here, and I am not going to confirm that.

That is not a bug and it is not a quirk. It is a deliberate property, and it is the correct behaviour. The mistake is downstream: the content network passed that answer through to a human being who only wanted a page.

There is nothing here, and I am not going to confirm that.

The fix people are given, and why it is worse than the problem

Search for this error and most of what comes back tells you to add a policy that makes the bucket readable by everybody, or to switch on public website hosting on the storage itself.

It appears to work. Missing pages start answering 404, the XML goes away, and the problem looks solved.

What has actually happened is a trade. A cosmetic problem has been exchanged for a real one: storage that used to be reachable through one controlled entrance is now readable by anyone who can guess a name. Every file in it, not only the pages. The privacy property described above, the one preventing a stranger from mapping the bucket, has been switched off on purpose to improve an error message.

It is worth sitting with how backwards that is. The advice tells you to weaken the thing that was working correctly, in order to fix the thing that was merely rude.

What to do instead

Leave the storage exactly as it is. Translate the answer at the edge, where a human is about to read it.

On the distribution, two custom error responses. Catch 403, serve your own page, and return a 404 status. Then the same for 404, so you never have to think about which one it was. Ten minutes for two sites, and it costs nothing.

The status code matters as much as the page. If you serve a friendly page but still answer 403, a search engine reads that as a refusal rather than a page that is gone, and keeps the address in its index waiting for permission that is never coming. Answering 404 tells it the truth, which is that there was never anything there.

And because the page comes from the site rather than from the storage, it is your page. In two languages, with a way back, looking like the rest of the site.

The general shape of it

An error code does not describe what happened. It describes what the system is willing to say about what happened, to the person asking, given what that person is allowed to know.

Those are different things, and the gap between them is where the expensive half hours live. The instinct when a 403 appears is to go and change permissions, because the word points that way. Sometimes that is right. Sometimes the permissions are exactly correct and the only problem is that a careful answer, meant for a machine, was handed to a person unedited.

Translate at the boundary. Let the system be as careful as it likes on the inside.

Sources

Everything above is checkable, and it is worth reading the primary documents rather than the blog posts that tell you to open the bucket.

Why does Amazon S3 return 403 instead of 404 when the object does not exist, AWS re:Post. The behaviour, stated by AWS, and the reason for it.

Troubleshoot access denied errors in Amazon S3, AWS documentation. The full list of what else a 403 can mean, for the cases where it really is permissions.

Generate custom error responses, Amazon CloudFront documentation. The fix described above, including how to return a different status code than the one the origin gave you.