09-10-2026 05:41 PM - edited 09-10-2026 05:42 PM
Hello, LiveCommunity team!
I created this post to share my experience with an issue on our branch PA-1420 NGFW, which is configured to act as both a DHCP server on the ethernet1/4 interface and a DHCP relay agent, but suddenly stopped working!
About 20 days ago, our DHCP server and DHCP relay agent stopped functioning as expected.
We use our PA NGFW at a critical branch office, where it acts as both a DHCP server and a DHCP relay agent, as shown in the diagram below:
PALO ALTO TOPOLOGY
Our internal clients complain that they cannot connect to the network, they have no internet access, and so on.
This issue arose suddenly on August 8, and we became aware of it when our customers informed us. We checked the entire path—including switches, servers, and the NGFW—but detected no errors or misconfigurations.
In the following section, you will see the troubleshooting steps, the RCA and the definitive solution.
Best Regards,
Daniel Romero
Senior Network/Security Engineer
PANW Partner
09-10-2026 06:35 PM - edited 09-10-2026 07:03 PM
Hello LiveCommunity Team!
Following up on the general description of the previous issue, I am sharing the troubleshooting steps, the root cause analysis (RCA), and the definitive solution, our client informs us that this issue occurred a few days ago.
TROUBLESHOOTING STEPS
1- DHCP DATAPLANE PACKET CAPTURE
On the NGFW we create a packet-capture filter on the Monitor > Packet Capture to capture the DHCP messages with the next filter:
PACKET CAPTURE FILTER 1
Source IP: Any Source Port: 68 (Used by DHCP Clients)
Destination IP: Any Destination Port: 67 (Used by DHCP Server/DHCP Relays)
Protocol: 17 (UDP)
PACKET CAPTURE FILTER 2
Source IP: Any Source Port: 67 (Used by DHCP Server/DHCP Relays)
Destination IP: Any Destination Port: 68 (Used by DHCP Clients)
Protocol: 17 (UDP)
We enabled packet capture on the data plane and observed that, during the *Receive* and *Firewall* stages, only DHCP Discovery packets were being received.
With this in mind, we confirmed that the NGFW was receiving the DHCP Discovery broadcast messages, thereby ruling out Layer 2 path issues (Aparently).
Next, we checked the configured DHCP pool and the allocation table, but found it empty; the NGFW was not offering any IP addresses to the DHCP clients.
2- DEBUG AND REVIEW THE ADVANCE DHCPD DAEMON LOGS
Based on the previous indications, we enable debug logging for the DHCP daemon (DHCPD) and have a DHCP client attempt to obtain an IP address; to do this, we use the following CLI commands:
NGFW> debug dhcpd global on dump
Next, we review the pan_dhcpd.log logs and see the following error messages:
NGFW PAN DHCPD ERROR LOGS
NGFW> less mp-log pan_dhcpd.log
pan_dhcpd_dp_connect(pan_dhcp_client_thread.c:106): Failed creating DHCP client socket
2026-08-15 20:09:32.489 -0500 Warning: pan_dhcpd_setup_dp_conn(pan_dhcp_client_thread.c:173): Failed to connect to DP
2026-08-15 20:09:32.489 -0500 Error: pan_dhcpd_setup_socket(pan_dhcpd.c:761): bind failed:(errno: 99) Cannot assign requested address
It's a best practice to restore the dhcpd debug level to info again.
NGFW > debug dhcpd global on info (Don't forget to do it)
We subsequently restarted the dhcpd process with the next command because critical internal clients were unable to access internal or external resources.
> debug software restart process dhcp core yes
After a few seconds, we restart the DHCP daemon, and suddenly, both the DHCP server and the DHCP relay agent are working exactly as expected!!
So we suspect there's a DHCP daemon bug, to confirm this we create a TAC support case and after a debug meet we received the next response:
"
Thank you for your patience while we investigated the recurring DHCP service interruptions on your PA-1420 firewall (PAN-OS 11.2.10-h3).
Our technical analysis of the pan_dhcpd.log file confirms that the error "errno: 99 (Cannot assign requested address)" is caused by a known internal software condition, specifically tracked under internal reference PAN-285181.
This issue involves a memory management limitation within the underlying network services stack that occurs after several days of continuous operation; this prevents the DHCP daemon from binding the communication sockets required to assign IP addresses via the DataPlane.
PAN-285181: Fixed an issue where the wifclient ran out of memory when Enhanced Application Logging was enabled and a sudden traffic increase caused a surge in EAL messages sent through WIF Client.
To use this fix, run the CLI command > debug iot eal memory-gc native
To provide a permanent solution and avoid the need to manually restart the daemon, we recommend the following course of action:
Software Update: The definitive fix for this resource allocation issue is included in PAN-OS version 11.2.10-h4 and later. We recommend scheduling the update to this version as soon as possible.
Post-update Optimization: Once the firewall is running the new version, a specific optimization command must be executed via the CLI to stabilize internal memory recovery:
NGFW> debug iot real memory-gc native
This command ensures that network processes manage memory efficiently, maintaining long-term stability for both DHCP server and DHCP relay functions.
We understand the impact this behavior has on your internal clients. Having identified the root cause and the specific solution, we have upgraded the priority of this case to "High."
We will continue to closely monitor your environment and remain available to assist you with the update process or any further validation you may require.
"
We have informed the client and are awaiting the final decision to begin the PAN-OS version upgrade.
I'm sharing the following commands and tips to resolve DHCP-related issues:
NGFW COMMANDS AND TIPS FOR DHCP ISSUES
To review the NGFW DHCP Monitor > System > ( description contains 'DHCP' ) or ( description contains 'dhcp' )
> show session all filter application dhcp (To review if exist any active session related to DHCP processes)
> show log system subtype equal dhcp direction equal backward (To review any DHCP traffic log)
> show system software status | match dhcp (To review the DHCP daemon status)
It's a best practice enable the DHCP Broadcast Session option under Device > Setup > Session Settings if your firewall is acting as a DHCP server, select this option to enable session logs for DHCP broadcast packets. If you do not enable this option, the firewall forwards the packets without creating logs for the DHCP broadcast packets.
I will update this discussion thread as soon as there is any news.
Thank you for your time, and I hope this information is helpful in your daily cybersecurity work. I would greatly appreciate your support by liking or accepting this as a useful post; it would help me a lot in becoming a CyberElite!
Best Regards,
Daniel Romero
Senior Network/Security Engineer
PANW Partner
09-10-2026 06:35 PM - edited 09-10-2026 07:03 PM
Hello LiveCommunity Team!
Following up on the general description of the previous issue, I am sharing the troubleshooting steps, the root cause analysis (RCA), and the definitive solution, our client informs us that this issue occurred a few days ago.
TROUBLESHOOTING STEPS
1- DHCP DATAPLANE PACKET CAPTURE
On the NGFW we create a packet-capture filter on the Monitor > Packet Capture to capture the DHCP messages with the next filter:
PACKET CAPTURE FILTER 1
Source IP: Any Source Port: 68 (Used by DHCP Clients)
Destination IP: Any Destination Port: 67 (Used by DHCP Server/DHCP Relays)
Protocol: 17 (UDP)
PACKET CAPTURE FILTER 2
Source IP: Any Source Port: 67 (Used by DHCP Server/DHCP Relays)
Destination IP: Any Destination Port: 68 (Used by DHCP Clients)
Protocol: 17 (UDP)
We enabled packet capture on the data plane and observed that, during the *Receive* and *Firewall* stages, only DHCP Discovery packets were being received.
With this in mind, we confirmed that the NGFW was receiving the DHCP Discovery broadcast messages, thereby ruling out Layer 2 path issues (Aparently).
Next, we checked the configured DHCP pool and the allocation table, but found it empty; the NGFW was not offering any IP addresses to the DHCP clients.
2- DEBUG AND REVIEW THE ADVANCE DHCPD DAEMON LOGS
Based on the previous indications, we enable debug logging for the DHCP daemon (DHCPD) and have a DHCP client attempt to obtain an IP address; to do this, we use the following CLI commands:
NGFW> debug dhcpd global on dump
Next, we review the pan_dhcpd.log logs and see the following error messages:
NGFW PAN DHCPD ERROR LOGS
NGFW> less mp-log pan_dhcpd.log
pan_dhcpd_dp_connect(pan_dhcp_client_thread.c:106): Failed creating DHCP client socket
2026-08-15 20:09:32.489 -0500 Warning: pan_dhcpd_setup_dp_conn(pan_dhcp_client_thread.c:173): Failed to connect to DP
2026-08-15 20:09:32.489 -0500 Error: pan_dhcpd_setup_socket(pan_dhcpd.c:761): bind failed:(errno: 99) Cannot assign requested address
It's a best practice to restore the dhcpd debug level to info again.
NGFW > debug dhcpd global on info (Don't forget to do it)
We subsequently restarted the dhcpd process with the next command because critical internal clients were unable to access internal or external resources.
> debug software restart process dhcp core yes
After a few seconds, we restart the DHCP daemon, and suddenly, both the DHCP server and the DHCP relay agent are working exactly as expected!!
So we suspect there's a DHCP daemon bug, to confirm this we create a TAC support case and after a debug meet we received the next response:
"
Thank you for your patience while we investigated the recurring DHCP service interruptions on your PA-1420 firewall (PAN-OS 11.2.10-h3).
Our technical analysis of the pan_dhcpd.log file confirms that the error "errno: 99 (Cannot assign requested address)" is caused by a known internal software condition, specifically tracked under internal reference PAN-285181.
This issue involves a memory management limitation within the underlying network services stack that occurs after several days of continuous operation; this prevents the DHCP daemon from binding the communication sockets required to assign IP addresses via the DataPlane.
PAN-285181: Fixed an issue where the wifclient ran out of memory when Enhanced Application Logging was enabled and a sudden traffic increase caused a surge in EAL messages sent through WIF Client.
To use this fix, run the CLI command > debug iot eal memory-gc native
To provide a permanent solution and avoid the need to manually restart the daemon, we recommend the following course of action:
Software Update: The definitive fix for this resource allocation issue is included in PAN-OS version 11.2.10-h4 and later. We recommend scheduling the update to this version as soon as possible.
Post-update Optimization: Once the firewall is running the new version, a specific optimization command must be executed via the CLI to stabilize internal memory recovery:
NGFW> debug iot real memory-gc native
This command ensures that network processes manage memory efficiently, maintaining long-term stability for both DHCP server and DHCP relay functions.
We understand the impact this behavior has on your internal clients. Having identified the root cause and the specific solution, we have upgraded the priority of this case to "High."
We will continue to closely monitor your environment and remain available to assist you with the update process or any further validation you may require.
"
We have informed the client and are awaiting the final decision to begin the PAN-OS version upgrade.
I'm sharing the following commands and tips to resolve DHCP-related issues:
NGFW COMMANDS AND TIPS FOR DHCP ISSUES
To review the NGFW DHCP Monitor > System > ( description contains 'DHCP' ) or ( description contains 'dhcp' )
> show session all filter application dhcp (To review if exist any active session related to DHCP processes)
> show log system subtype equal dhcp direction equal backward (To review any DHCP traffic log)
> show system software status | match dhcp (To review the DHCP daemon status)
It's a best practice enable the DHCP Broadcast Session option under Device > Setup > Session Settings if your firewall is acting as a DHCP server, select this option to enable session logs for DHCP broadcast packets. If you do not enable this option, the firewall forwards the packets without creating logs for the DHCP broadcast packets.
I will update this discussion thread as soon as there is any news.
Thank you for your time, and I hope this information is helpful in your daily cybersecurity work. I would greatly appreciate your support by liking or accepting this as a useful post; it would help me a lot in becoming a CyberElite!
Best Regards,
Daniel Romero
Senior Network/Security Engineer
PANW Partner
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
