How does Expanse know about my website?

[SnappyRcx]

I have the following message in the User Agent field of my Apache log:

Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com"

How does Expanse know to scan my website?

Here's why I'm asking:

• The subdomain is brand new (automatically generated just a few hours ago)
• The subdomain is not advertised anywhere
• The only way to get to this Apache is by specifying the exact domain in the Host HTTP header
• The site sents 'noindex', 'nofollow' and related headers for every request to prevent indexing
• As far as we can tell, we don't have any malware on our computers.

Just two people know about this subdomain.

So... which software spied on who and how?

The subdomain most likely only appeared in a Firefox browser for one of us before Expanse and a bunch of other scanners started accessing the site

[BPry]

@SnappyRcx,

Two things

1. As soon as you register a public DNS record everyone can find it. There's services out there that actively sell access to newly registered domains for a minimal amount of money that aggregate everything for you; I have to image that PAN both has their own service they've created for this and likely actively buy access to this data said through aggregators as well.
2. As soon as someone using a PAN firewall (potentially even you) visits an unknown URL the firewall is telling PAN that it needs to be analyzed because it's unknown. So in the event that you have a PAN device in your network it's quite possible it was you telling PAN that it exists and needs to be analyzed. 

[SnappyRcx]

@BPry

Option 1 doesn't apply, because the subdomain is resolved via a wildcard DNS resolver. It's not itself present in DNS, and it's a subdomain of a another registered domain.

For Option 2 though, after a some brief brainstorming with my colleague, I think I have a candidate shortlist: either Let's Encrypt, or Certificate Transparency logs. My website uses Let's Encrypt for its SSL. All other variables appear to be controlled for: it's my container, behind my reverse proxy, built on my Docker image, using my PHP code and everything is, to the best of my knowledge, not blabbing to outside services. The only places where the subdomain is known are: Traefik, Let's Encrypt, CT logs. The container doesn't know its subdomain beforehand.

I could validate this hypothesis at some point be also replacing the PHP code with some hello world HTML.

I was just surprised, because this site is only meant for our temporary software development purposes.