PA-7000 Not passing syslog traffic to Tufin

[netzwerk-admin]

Hi All,

We have a PA-7000 (7.1) and Tufin (for syslog).

The system was previously setup to forward syslog traffic to Tufin. 

Then all of a sudden, Tufin wasn't receiving any traffic.

What I have done so far:

1. Went through the saved configurations to see when the syslog config was changed.
   • From the saved configs, I could not see anything that was changed that affected syslog forwarding.
   • No Palo Alto or Tufin updates were installed.
2. Rechecked the syslog forwarding configuration (at least 5 times as of this writing).
3. Ran tcpdump on Tufin server 
   • traffic was not getting to Tufin
   • 14:00:54.560060 IP (tos 0x0, ttl 60, id 62901, offset 0, flags [DF], proto UDP (17), length 358)
     10.63.249.5.43067 > tufina01.syslog: [udp sum ok] SYSLOG, length: 330
     Facility user (1), Severity error (3)
     Msg: Dec 28 14:02:06 fw-f-wm-dc-1c.infra.dvag.com 1,2017/12/28 14:02:06,010108000926,SYSTEM,userid,0,2017/12/28 14:02:06,,connect-agent-failure,,0,0,general,high,"TS-Agent Citrix wpsxaaabn02.id(vsys1): Error: Failed to connect to wpsxaaabn02.id(10.61.85.151):5009 details: none",827871,0x0,0,0,0,0,,fw-f-wm-dc-1c
   • As seen above, only system type information is reaching Tufin
4. Ran tcpdump on  PA-7000
   • 12:14:03.004652 IP 10.63.249.5.53918 > 10.63.98.59.syslog: SYSLOG user.info, length: 226
     12:14:20.101956 IP 10.63.249.5.35845 > 10.63.98.59.syslog: SYSLOG user.info, length: 227
     12:14:31.557722 IP 10.63.249.5.37782 > 10.63.98.59.syslog: SYSLOG user.info, length: 344
     12:14:31.573796 IP 10.63.249.5.53918 > 10.63.98.59.syslog: SYSLOG user.info, length: 253
     12:14:31.640424 IP 10.63.249.5.35845 > 10.63.98.59.syslog: SYSLOG user.info, length: 242
     12:14:32.604810 IP 10.63.249.5.37782 > 10.63.98.59.syslog: SYSLOG user.info, length: 344
     12:14:32.616503 IP 10.63.249.5.53918 > 10.63.98.59.syslog: SYSLOG user.info, length: 253
     12:14:32.682839 IP 10.63.249.5.35845 > 10.63.98.59.syslog: SYSLOG user.info, length: 242
   • Traffic Monitoring shows that syslog (udp 514) packets are allowed, Session End Reason 'aged-out'

If someone could push me in the right direction to correct this I would greatly appreciate it.

Regards,

Jasper Freeman

[BPry]

@netzwerk-admin,

Just out of curiosity have you attempted to restart the management plane since you began experiancing these issues? 

[netzwerk-admin]

No we haven't.

I'll give it a try next week. Don't want to make any changes on a Friday. Especially before a long weekend.

Jasper

[BPry]

@netzwerk-admin,

If it doesn't work then let us know, but I would assume that this should get things working correctly again. 

[netzwerk-admin]

Well, scratch that.

A colleague said the system was restarted 8 days ago because a security bug.

So, that answer is yes, the management plane was restarted.

[BPry · Accepted solution]

@netzwerk-admin,

So that would indicate that this issue is at least more than 8 days old, and didn't start with the restart? 

A couple things that I would check. 

1) Verify that nobody removed the log-forwarding profile from your security policies. I've seen this happen in the pass with multiple firewall admins. 

2) Verify that you can actually get a response from Tufin and that there isn't a routing issue. I would expect the Session End Reason to show as 'aged-out' on Syslog traffic, as the firewall never gets anything to tell it to close the session.

Is Tufin functioning for other devices okay? 

[netzwerk-admin]

Yes, Tufin is functioning for other devices.

I just restarted the monitoring for the Palo Alto on Tufin and now I'm seeing that syslog traffic is arriving at the Tufin interface.

But, I'm a little confused. If I execute tcpdump it says it is doing the dump an eth0

     e13itfd@fw-f-wm-dc-1c(active)> tcpdump filter "src 10.63.249.5 and port 514" snaplen 0
     Press Ctrl-C to stop capturing

     tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

Results:

             e13itfd@fw-f-wm-dc-1c(active)> view-pcap mgmt-pcap mgmt.pcap
             07:33:17.762228 IP 10.63.249.5.43067 > 10.63.98.59.syslog: SYSLOG user.error, length: 330
             07:34:15.498597 IP 10.63.249.5.46508 > 10.63.98.59.syslog: SYSLOG user.error, length: 316
             07:35:40.596228 IP 10.63.249.5.38662 > 10.63.98.59.syslog: SYSLOG user.error, length: 316
             07:37:00.204343 IP 10.63.249.5.43067 > 10.63.98.59.syslog: SYSLOG user.error, length: 330
             07:37:35.926424 IP 10.63.249.5.46508 > 10.63.98.59.syslog: SYSLOG user.error, length: 330
             07:37:35.926518 IP 10.63.249.5.38662 > 10.63.98.59.syslog: SYSLOG user.error, length: 329
             07:37:40.205790 IP 10.63.249.5.43067 > 10.63.98.59.syslog: SYSLOG user.error, length: 330
             07:39:09.685971 IP 10.63.249.5.37782 > 10.63.98.59.syslog: SYSLOG user.info, length: 265
             07:39:09.685980 IP 10.63.249.5.35845 > 10.63.98.59.syslog: SYSLOG user.info, length: 277

What confuses me is that on the Palo Alto I don't see any TRAFFIC labled packets. On Tufin a tcpdump with the src=10.63.249.5 also shows no packets at all.

Then I decided to see if there are syslog traffic being sent on the Log Card IP. Tufin is seeing syslog TRAFFIC from the Log Card IP; however, the Palo Alto shows no TRAFFIC at all.

This is confusing.

@BPry, thanks for the help.

[BPry]

@netzwerk-admin,

By log card do you mean the SMC? 

[netzwerk-admin]

@BPry

Actually, it's one of the interface on the NPC-20GQ module.

For example, we have ethernet1/3 (type: Log Card) configured for passing log information to Tufin.