Intermittent Group Membership problem

[rds]

We are currently having a problem with a new domain where the group membership intermittently disappears.

If you run the command "show user user-IDs match-user domain\" (4.1.x) or "show user pan-agent user-IDs match-user domain\" (4.0.x) it shows users mapped to AD groups.

This is happening on a single new domain where all other domains are working fine with the same configurations.

It is also happening for the same problem domain in 6 different regions where different firewalls and pan-agents are involved.

Some regions we are running PANOS 4.0.11 on the firewall and others we are running 4.1.9. All pan-agents are on 3.1.2.

We have tested upgrading the pan-agents as recommended by PA support in the regions where we are running 4.1.9 but this makes no difference.

So the group memberships disappears whether the pan-agent is handling the group membership (3.1.2) or the Firewall (4.1.5-6).

The user-ip mappings and group membership is learnt ok and after a period of time usually about an hour, the group memberships disappears for around 10 minutes sometimes more.

The user-ip mappings are not affected, only group membership, and there is nothing in the logs to indicate a problem either on the firewall or the pan-agents.

When the group memberships disappears the command "debug user-id reset user-id-agent" brings them back but they only disappear again after about an hour.

We have checked the service account permissions (able to read security event logs and write to local disk & registry etc.)

It seems to be something relating to this domain as all other domains are working fine.

If any anybody has any ideas please let me know, seems to have PA support stumped too and we are out of ideas?

[mikand]

How many DC-servers do you have and how many PAN-agents?

And are each PAN-agent monitoring several servers at once or is it a 1:1 relation between each DC-server and each PAN-agent-server?

Because if you have a 1:1 relation (either running PAN-agent locally on the DC-server and configure it to only follow log from localhost or run PAN-agent on a dedicated server but configure it to only follow one DC-server (the one which is closest to it networkwise)) you will not only lower the WAN-bandwidth of security logs being copied back and forth but also easier to identify which region is faulty.

I would also again verify the settings of each PAN-agent service, do you perhaps have some incorrect white/blacklist on one of these servers or so? Or for that matter enabled netbios or WMI lookups (im thinking if one of the PAN-agents isnt allowed to query the clients either by firewalls in your infrastructure or by local firewalls on the clients - or for that matter a MTU-problem)?

What about the useraccount which runs the PAN-agent service?

[mikand]

By the way, could this be the problem (just saw it in the release notes for PANOS 5.0.2)?

"

* 47280, 46424, 45635 – The User-ID agent on a Windows 2008 server was intermittently failing to respond when the directory contained 50,000+ users, causing valid user to IP mapping information to be deleted on the firewall. This occurred when the session limit of the firewall was being reached. Issue was due to a buffer problem that occurred when trying to write the user to IP mapping to the firewall. In order for this fix to work properly, the User-ID agent must be at 5.0.1 or later.

"

[rds]

We generally have 2 pan-agents (one as a backup) in each region on dedicated servers only monitoring the DC's in the local region.

Pan-agent settings are identical to all other agents for domains which are working ok. Netbios probing is enabeld by default.

User account running the pan-agent service has all the correct permissions.

Also not sure it's related to your second post as there are no where near 50,000 users in the directory, the user to IP mappings do not get deleted (only the AD group membership) and our session limit on our busiest firewall is under 30,000.

One other thing to mention is that this new domain (2008) which is having the problem belongs to a business unit that is migrating from one of the existing domains (2003). And what we've noticed is that after further testing yesterday is that when we point our lab firewall to an agent on the new domain it seems to keep the group membership. As soon as we add an agent on the old domain the new domain membership disappears.

Looks like there may be some issue between the 2 domains? the only things I've noticed so far is that the AD group names are the same.

There is also a trust between these domains although that goes for most of our other domains too.

eg. group in old domain is "abc\g-abc palo alto group 1" and group in new domain is "bc\g-abc palo alto group 1"

As they are busy migrating we can't delete the old agents yet but that would be the next test.