topic Re: url filtering flow in General Topics

url filtering flow

Roby_Sreejith — Tue, 23 Aug 2016 14:57:21 GMT

The url filterng flow is like below:

data plance -> management plane -> cloud.

I am seeing the category of ome sites is different than test url site of palo alto.

For Ex:

In Palo alto firewall below command gives this output

show running url xxx

malaware

which means the the ur category in data plane is malware.

However when i run

test url xxx

web-advertisements (Base db) expires in 1200 seconds
web-advertisements (Cloud db)

Which means the url category is different in management plane and cloud. That is the reason the website to test urls in palo alto also gives correct category.

I ran " clear url-cache url xxx"

then it reset the category in dataplane to " not resolved" I beleive it is expected in PAN DB.

show running url xxx

not-resolved expires in 0 seconds

After some minutes( it took more than 5 min) the

show running url

web-advertisements expires in 301 seconds

My questions are:

How the category updated in data plane and management plane?

What is the reason 2 different category in data lane and managment plane?

Why it took more than 5 to 10 min to update the category in data plane? How the category go updated in data plane after clearing the url.

If I have not cleared the dataplane url category, the website should have blocked always. What is the best recommeneded setups to keep correct category across dataplane as the url filtering checks data plane first.

What is the time out value of data plane cache entries? Do we have to clear manually using command always?

Will enabling dynamic db will help?

Re: url filtering flow

Roby_Sreejith — Wed, 24 Aug 2016 07:01:35 GMT

any help

Re: url filtering flow

reaper — Wed, 24 Aug 2016 07:46:25 GMT

Hi Robby

An URL may get cached when it is still categorized as CatA, but if this is a popular url inside your organization, the cache may keep getting refreshed on-device while the cloud categorization is changed to CatB in the meanwhile. your cache will still hold CatA

you can try clearing the URL from your cache to force a fresh cloud lookup:

admin@myNGFW> clear url-cache 
> all   Clear all URLS in data plane
> url   Clear the specified URL from data plane

Re: url filtering flow

Roby_Sreejith — Wed, 24 Aug 2016 09:26:37 GMT

When I run the command on firewall i am get 0 seconds as expiry

show running url xxxx.xx

xxxxx.xx malware expires in 0 seconds

However on some firewall i am getting expiry in some specified seconds:

show running url xxxxx.xx

xxxx.xx web-advertisements expires in 906 seconds

why on firewall i am getting 0 sec expiry for cacahe and for other 906 ec expiry.

0 means will it never expire?

what settings causes these difference on both firewall.

Re: url filtering flow

reaper — Wed, 24 Aug 2016 09:55:15 GMT

i'm not sure

have you tried clearing the cache ? maybe the url has gotten stuck somehow

you could try restarting the device server, in case it is in a state that causes it to not clear it's cache:

> debug software restart process device-server

Re: url filtering flow

Roby_Sreejith — Wed, 24 Aug 2016 11:31:15 GMT

After clearing cache , it is started shoiwng correct category and exiry timers are reducing correctly.

It is not 0 anymore.

I was wondering before clearing what might have happened. Why it was stuck at 0.

Re: url filtering flow

Roby_Sreejith — Wed, 24 Aug 2016 14:41:31 GMT

I have taken an url which is not accessed over the firewall traffic.

When i checked dataplane cacahe i can see:

xxxxx not-in-url-cache

Then I ran debug dataplane test url-resolve-path command

Then it started showing correct category and expiry interval is started showing as 1800 sec.

news expires in 1797 seconds

I have waited for 1800 sec.

Now the data plane cacahe results says

xxxxxx news expires in 0 seconds

This is stuck there. I mean the category is not expirying. the category stays even after 1800 sec.

I am sure the website is not accessed again in firewall traffic.

My question, why the category is not clearing from data plane cacahe after 1800 sec

Re: url filtering flow

reaper — Wed, 24 Aug 2016 15:07:39 GMT

Try restarting the device server service, it may be stucjk and unable to clear the entries

Re: url filtering flow

Roby_Sreejith — Fri, 26 Aug 2016 06:02:44 GMT

My question, why the category is not clearing from data plane cacahe after 1800 sec

This is stuck there. I mean the category is not expirying. the category stays even after 1800 sec.

I am sure the website is not accessed again in firewall traffic.

see this

xxxxxx news expires in 0 seconds

it is stuck for 2 to 3 days.

I have tried this clearing url in mutple firewalls. it is also giving same results. it seems like as per design but require some exlanation