Why they always exist
A staging site gets created for a specific piece of work. A redesign, a migration, a client preview, a plugin someone wanted to test without breaking production. That work finishes.
Removing the staging site is nobody's task. It was not in the brief, it does not block anything, and the person who created it has moved to the next job. It sits there costing nothing and doing nothing, until it is four years old and running software that has not been patched since it was installed.
What is actually on them
This varies enormously and is worth ten minutes before deciding how urgently to act.
- Placeholder content and nothing else. Untidy, low risk, remove it when convenient.
- A copy of the production database, taken at the time so the redesign could be tested against real content. Now a customer data exposure with no access control.
- An admin login sharing credentials with production, on software several years out of date.
- An API or internal tool that was never meant to face the public and has no authentication because it was on a trusted network at the time.
Why they are trivially findable
Certificate transparency logs are public and complete. Every certificate ever issued for any name under your domain is listed, which includes the staging site, because whoever set it up quite reasonably gave it a certificate.
Nobody needs to guess names or scan your servers. The list is published, permanently, and searching it takes seconds. That is how we find them and it is how anyone else finds them.
Restricting one
Fastest fix is taking it offline, and most of these are not in use. If it is in use, basic authentication in front of it is one line of configuration and stops both crawlers and casual visitors. Restricting by IP address is better still.
A robots.txt file is not a control. It asks well-behaved crawlers not to index and does nothing whatever to stop a person reading the site.
The pattern underneath
This is the same shape as dangling records and unauthorised mail senders. Something is created for a good reason, the reason ends, and removal is a separate action that belongs to nobody. Every one of these is an organisational gap wearing technical clothes.