How to find ALL the links related to a website.

[JANARTHANAN1392]

Looking for the better idea.

User reported xyz.com site is not working, When I look at the firewall logs xyz.com was allowed in firewall.

And I filtered the logs with source IP for 5mins time frame and found deny logs for particular IP address 1.1.1.1

I am not sure what is the url for 1.1.1.1, In such case what you will do to find the URL for 1.1.1.1

I usually do a search in splunk log server, for this IP 1.1.1.1 it will show all the related logs, I filter with dns logs then I can find the hostname for 1.1.1.1 ex. ab.xyz.com, Now I am able to permit this additional URL, it will fix the issue.

What if I don't have splunk log server, how can I identify this additional link(ab.xyz.com)

Also is there a way to find all the links that is being used in a website?

#suggestion

[PavelK]

Hello @JANARTHANAN1392

thanks for the post in LIVEcommunity!

Right of the hand I can think off using Unified log under Monitor > Logs > Unified. Unified logs is an aggregation of all logs (Traffic, Threat, URL,..). This should provide a single view of blocked traffic regardless its log origin. 

Kind Regards

Pavel

[JANARTHANAN1392]

@PavelK 

Thanks for the response.

even unified log, it gives the denied IP traffic logs not it's hostname details for the denied IP. Not able to find the hostname for that IP.

How can I find it.

Also is there a way to find all the links that is being used in a website?

[aleksandar.astardzhiev]

Hi @JANARTHANAN1392 ,

Are you using 1.1.1.1 just example of this is the actual address you see the logs? I am asking because 1.1.1.1 is Cloudflare public DNS, so I would expect to see deny traffic, if you don't allow any DNS, but use only internal/trusted DNS

Now on the question - URL log is created only by URL filtering profile if blocked or allowed. If traffic is blocked by security rule, traffic log will be generated. But if you remember from documents - deny security rule does not apply any profile, so traffic is denied before being inspected by url filter to create any url log.

So it is Catch 22. The sad think is that if you have custom URL category as match criteria in security rule, you still don't have a way to know which url was blocked and which was allowed.

I recently found myself in similar situation - I had to found all the URL that were required and allow only them, blocking everything else. But user was not able to provide exact list. I would suggest you two options:

- Create temp rule for your test user allowing any traffic (you probably want to filter http and https ports at minimum). Apply URL filtering profile that has "alert" action for all categories. Ask the user to test and open the problematic page. Check the URL filtering logs take a note for all of the URLs that FW has detected.

- At the very bottom of your policy, right above the "cleanup" rule create new rule that allow any destination for ssl and web-browsing. But this time apply URL filtering profile which set all categories to block. Essentially this rule will still act as "deny everything that is not explicitly blocked", but since the URL filtering will block the traffic and not the security rule, you will have URL log (instead of traffic log), which will give you information for all URL that FW has blocked, because they were not explicitly allowed.