Episode Transcript:
John:
Hello PANCasters, today we have Olivier again to discuss how to troubleshoot log forwarding to Cloud Services. Welcome back Olivier.
Olivier:
Hello John, thank you for having me back on PANCast.
We are having quite some cases about issues related to logs forwarding to the Cloud Services, so I wanted to discuss about few troubleshooting steps to perform.
John:
Great, so what are the use cases where our next gen firewalls would send data to Cloud Services?
Olivier:
So first of all, if you are using AIOps, Strata Cloud Manager, IoT Security, Cortex Data Lake, you may be interested in this
Olivier Zheng, PCNSE, is a Staff Support Engineer at Palo Alto Networks. As SME Management/Logging Reporting in Technical Assistance Centre Singapore, he is supporting customers and participating in multiple knowledge sharing initiatives by writing content in the Knowledge Base, by delivering training to internal engineers. He is responsible for 1 issued patent. Olivier holds a Master of Science Mobile and High Speed telecom networks from Oxford Brookes University, UK and a Master of Science in Computer Science and Information Technology from ESI SUPINFO Paris, France. episode.
Basically, any existing or upcoming cloud services which parse the firewall logs, the telemetry data are using the same common points:
- The device certificate
- The Hub to bind devices to application
- And the log receiver
John:
OK so let’s start with the device certificate.
Olivier:
For the sake of simplicity, I will just talk about the device certificate as it becomes the certificate used to authenticate the device from PAN-OS 10.1 and above.
If you read somewhere “thermite” certificate, it is that certificate I am talking about.
To start, the device certificate requires the firewall to access Palo Alto networks servers.
So you can make sure the network is not blocking the access to our servers. How? On the firewall, you can check the traffic logs if anything to the server IP is allowed or not. You can also do a packet capture but usually you shouldn’t have to perform such detailed troubleshooting. For reference, you can find in the transcript the FQDNs required to get the device certificate.
Another action you can take is to simply try to re-fetch a new device certificate. If your device cannot get its device certificate and you are pretty confident the network part is not blocking the required traffic, contact the TAC so we review and clear our backend if needed.
John:
So if the device certificate is present and it is valid, what can we check on the Hub?
Olivier:
So the Hub is the portal with the cloud services available to your account. If you never heard of it, open your browser, go to apps.paloaltonetworks.com and log on with your CSP account.
Probably there will be nothing if it is your first time connecting to it. Otherwise, you should see the different apps available.
The first thing you can check is to make sure the Hub is displayed on the TSG aware mode aka Hub2.0. Except if there is an application you are using which is not TSG aware, you should be in this Hub2.0 mode, as it is safe to assume all applications will be TSG aware.
So you see the background color of the portal : if it is dark blue, you are good you are on the Hub2.0. If it is not, you simply have to switch the toggle on the top left of the portal to switch from Hub1.0 to Hub2.0.
I will clear some misconceptions here.
Misconception#1:Being superuser on the CSP does not mean your account will be superuser on the Hub.
First you need a TSG, which is a logical container where you associate devices, apps, users. So if there is no TSG, it is normal there is nothing to see.
If there is a TSG, you should check with your colleague if someone has created a TSG with an app. And that is because of the misconception #2 - TAC do not have administrative rights on customer accounts, so if you have any access issue to a TSG / app, you may resolve the issue faster by asking your colleagues than opening a case to TAC.
Once you have ensured that there is no user right related issue, the next check on the Hub is to simply make sure the devices are attached to the TSG, and if needed that the devices are associated with the apps, usually it is the apps requiring a license.
For instance, AIOps Free is applied by default to all devices as there is no license requirement. But IoT Security require the device to be associated with the app so the license can be associated to the device.
Besides those 2 common issues, if there is something wrong on the Hub, open a case to TAC with at least the TSG number, and a screenshot of the issue.
John:
Great, what about the log receiver?
Olivier:
At that point, your firewall has its device certificate, everything is clean on the Hub side, so we need to focus on the actual log forwarding.
So on the firewall (yes, because currently most of the use cases are using logs from firewalls), make sure all the connections are initiated from the same interface, by default it is the management interface.
But it happens that you may decide to use service routes to use a specific dataplane interface for the communication with Palo Alto Networks’ log receiver. So you will have to put a service route for all the required region specific FQDNs to use the same interface.
Another thing to make sure is the region configured on the device, it needs to match with the region of the deployed app on the Hub : if you have CDL in the Europe region, and the firewall is sending to the Singapore region, it will not work.
Finally, if the traffic is sent out from the right interface and the region is correct, you need to make sure the devices in transit is not blocking the traffic and it is correctly forwarding it to the internet. You can review the traffic logs for that on the firewall if you are looping back the management interface to the dataplane for instance, or you can check on the next device what is happening to the traffic.
You can also get the device_telemetry_curl.log log file output from the CLI, you can see the transfer output and see to which URL the issue is currently happening.
John:
Thanks Olivier, is that all we need to know?
Olivier:
Yes, I cannot share all my troubleshooting tricks. But if you check those different points, you have cleared most of what I would check.
John:
Aha ok, so what are the takeaways for today?
Olivier:
So to sum up, I would say:
- Make sure the device cert is present on the firewall
- Make sure the device is correctly associated to the app in the TSG
- Make sure the traffic is flowing as expected
John:
Thanks again Olivier. PANCasters, as always the transcript will be on live.paloaltonetworks.com and remember to subscribe to PANCast on your favorite podcast platform.
