url filtering flow

[Roby_Sreejith]

The url filterng flow is like below:

data plance -> management plane -> cloud.

I am seeing the category of ome sites is different than test url site of palo alto.

For Ex:

 In Palo alto firewall below command gives this output

show running url xxx

malaware

which means the the ur category in data plane is malware.

However when i run 

test url xxx 

web-advertisements (Base db) expires in 1200 seconds
web-advertisements (Cloud db)

Which means the url category is different in management plane and cloud. That is the reason the website to test urls in palo alto also gives correct category.

I ran " clear url-cache url xxx"

then it reset the category in dataplane to " not resolved" I beleive it is expected in PAN DB.

show running url xxx

not-resolved expires in 0 seconds

After some minutes( it took more than 5 min) the 

show running url

web-advertisements expires in 301 seconds

My questions are:

How the category updated in data plane and management plane?

What is the reason 2 different category in data lane and managment plane?

Why it took more than 5 to 10 min to update the category in data plane? How the category go updated in data plane after clearing the url.

If I have not cleared the dataplane url category, the website should have blocked always. What is the best recommeneded setups to keep correct category across dataplane as the url filtering checks data plane first.

What is the time out value of data plane cache entries? Do we have to clear manually using command always?

Will enabling dynamic db will help?

[Roby_Sreejith]

any help

[reaper]

Hi Robby

An URL may get cached when it is still categorized as CatA, but if this is a popular url inside your organization, the cache may keep getting refreshed on-device while the cloud categorization is changed to CatB in the meanwhile. your cache will still hold CatA

you can try clearing the URL from your cache to force a fresh cloud lookup:

admin@myNGFW> clear url-cache 
> all   Clear all URLS in data plane
> url   Clear the specified URL from data plane

[Roby_Sreejith]

When I run the command on firewall i am get 0 seconds as expiry

show running url xxxx.xx

xxxxx.xx malware expires in 0 seconds

However on some firewall i am getting expiry in some specified seconds:

show running url xxxxx.xx

xxxx.xx web-advertisements expires in 906 seconds

why on firewall i am getting 0 sec expiry for cacahe and for other 906 ec expiry.

0 means  will it never expire?

what settings causes these difference on both firewall.

[reaper]

i'm not sure

have you tried clearing the cache ? maybe the url has gotten stuck somehow

you could try restarting the device server, in case it is in a state that causes it to not clear it's cache:

> debug software restart process device-server

[Roby_Sreejith]

After clearing cache , it is started shoiwng correct category and exiry timers are reducing correctly.

It is not 0 anymore.

I was wondering before clearing what might have happened. Why it was stuck at 0.

[Roby_Sreejith]

I have taken an url which is not accessed over the firewall traffic.

When i checked dataplane cacahe i can see:

xxxxx not-in-url-cache

Then I ran debug dataplane test url-resolve-path  command

Then it started showing correct category and expiry interval is started showing as 1800 sec.

news expires in 1797 seconds

I have waited for 1800 sec. 

Now the data plane cacahe results says

xxxxxx  news expires in 0 seconds

This is stuck there. I mean the category is not expirying. the category stays even after 1800 sec. 

I am sure the website is not accessed again in firewall traffic.

My question, why the category is not clearing from data plane cacahe after 1800 sec

[reaper]

Hi

Try restarting the device server service, it may be stucjk and unable to clear the entries

[Roby_Sreejith]

My question, why the category is not clearing from data plane cacahe after 1800 sec

This is stuck there. I mean the category is not expirying. the category stays even after 1800 sec. 

I am sure the website is not accessed again in firewall traffic.

see this 

xxxxxx  news expires in 0 seconds

it is stuck for 2 to 3 days.

I have tried this clearing url in mutple firewalls. it is also giving same results. it seems like as per design but require some exlanation